I’d appreciate hearing whether anyone using the PIRUS plugin has seen anything similar to the following, or from anyone else with wisdom to offer …

I have noticed that netstat on our eprints service shows many connections to 54.72.175.35:80 in CLOSE_WAIT state, originating from instances of the httpd process (see below) and that these connections take a long time to get to LAST_ACK and then stay in that state for a long time. If I understand what I read correctly, CLOSE_WAIT occurs when the remote end has requested that the TCP connection be closed (sent a FIN packet), to which the local protocol stack has responded with an ACK, and is then waiting for the local process with the connection open to close it.

54.72.175.35 is an Amazon AWS VM that appears to host a number of services but it appears most likely that the service of relevance here is http://www.jusp.mimas.ac.uk/ and what we’ve got is the PIRUS plugin reporting each full-text download to the JUSP COUNTER application.

So it looks as to me as though the plugin is failing to close() the connection promptly on receipt of a close request from the JUSP COUNTER application.

I’m a dilettante when it comes to Perl and eprints code but, from glancing at the plugin code, I cannot see anything obviously amiss, so I’m guessing that the answer lies inside LWP::UserAgent or LWP::ConnCache as used by the plugin.

Does anyone else recognise this behaviour or have any suggestions on how to fix it, or can tell me I’m barking up the wrong tree?

This came to light because on a few occasions recently the NERC eprints service has become completely unresponsive with connections hanging and timing out and the logs recording many HTTP 500 errors and “Software caused connection abort at /opt/eprints3/perl_lib/EPrints/Page.pm line 78.\n”.

From the netstat output I guessed that we were hitting the limit on the number of httpd processes and was able to recover by stopping Apache, waiting until the connections cleared, then restarting it. With this done, eprints springs back into life.

Thanks,

            Alan.

 

Alan Cox | Infrastructure Team NERC Research Technology Services Polaris House, North Star Avenue, Swindon, SN2 1EU, UK Tel: +44 (0)1793 411963 | Email: agc@nerc.ac.uk NERC | Planet Earth Online | Follow @NERCscience & @NewOnNORA

 

 

netstat output extract:

 

tcp        1      1 139.166.209.11:45805        54.72.175.35:80             LAST_ACK    -                  

tcp        1      0 139.166.209.11:45810        54.72.175.35:80             CLOSE_WAIT  12784/httpd        

tcp        1      0 139.166.209.11:45822        54.72.175.35:80             CLOSE_WAIT  12704/httpd        

tcp        1      0 139.166.209.11:45820        54.72.175.35:80             CLOSE_WAIT  12633/httpd        

tcp        1      0 139.166.209.11:45768        54.72.175.35:80             CLOSE_WAIT  11758/httpd        

tcp        1      0 139.166.209.11:45785        54.72.175.35:80             CLOSE_WAIT  11773/httpd        

tcp        1      0 139.166.209.11:45856        54.72.175.35:80             CLOSE_WAIT  12497/httpd        

tcp        0      0 139.166.209.11:45870        54.72.175.35:80             ESTABLISHED 12217/httpd        

tcp        1      0 139.166.209.11:45866        54.72.175.35:80             CLOSE_WAIT  10528/httpd        

tcp        1      0 139.166.209.11:45864        54.72.175.35:80             CLOSE_WAIT  12620/httpd        

tcp        1      0 139.166.209.11:45865        54.72.175.35:80             CLOSE_WAIT  11561/httpd        

tcp        1      0 139.166.209.11:45835        54.72.175.35:80             CLOSE_WAIT  11774/httpd        

tcp        1      0 139.166.209.11:45832        54.72.175.35:80             CLOSE_WAIT  12785/httpd        

tcp        1      0 139.166.209.11:45833        54.72.175.35:80             CLOSE_WAIT  12780/httpd        

tcp        1      1 139.166.209.11:45846        54.72.175.35:80             LAST_ACK    -                  

tcp        1      0 139.166.209.11:45844        54.72.175.35:80             CLOSE_WAIT  12605/httpd        

tcp        1      0 139.166.209.11:45842        54.72.175.35:80             CLOSE_WAIT  8836/httpd         

tcp        1      1 139.166.209.11:45843        54.72.175.35:80             LAST_ACK    -                  

tcp        1      1 139.166.209.11:45840        54.72.175.35:80             LAST_ACK    -                  

tcp        1      0 139.166.209.11:45841        54.72.175.35:80             CLOSE_WAIT  12701/httpd        

tcp        1      0 139.166.209.11:45854        54.72.175.35:80             CLOSE_WAIT  12783/httpd        

tcp        1      0 139.166.209.11:45852        54.72.175.35:80             CLOSE_WAIT  12632/httpd        

tcp        1      0 139.166.209.11:45853        54.72.175.35:80             CLOSE_WAIT  12781/httpd        

tcp        1      0 139.166.209.11:45850        54.72.175.35:80             CLOSE_WAIT  12777/httpd        

tcp        1      0 139.166.209.11:45848        54.72.175.35:80             CLOSE_WAIT  12789/httpd        

tcp        1      0 139.166.209.11:45849        54.72.175.35:80             CLOSE_WAIT  12782/httpd