EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #00613

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: Eprints Server Freezes Often

I think you can't do much other than:
a - restart via cronjob the apache and mysql process, every night - this help 
b - find why you've so much connections and from where
c - limit who is abusing from the service

for b, you can use the logs.

for c, you can use mod_bw from Apache. In debian just do this:

apt-get install libapache2-mod-bw
and add this to /usr/share/eprints3/archives/<your-archive>/var/auto-apache.conf (remember to do this when you re-run regeneration of apacheconf with epadmin tools) 

        # turn bw limitation on
        BandwidthModule On

        # force limitation on every request
        ForceBandWidthModule On
              BandWidth all 0
              MaxConnection all 20


inside the before </VirtualHost>
in this way abusers can connect only 20 times and not infinite. Usually they throttle or stop to abuse you :)
Try also the mysql tuning, they help expecially the thread_cache_size and table_cache. 


Il 22/05/2012 14:30, Francis Jayakanth ha scritto:

On Tue, 22 May 2012, Yuri wrote:


Il 22/05/2012 08:43, Francis Jayakanth ha scritto:

On Mon, 21 May 2012, Yuri wrote:


Lower the number of child and forked processes in apache config. The
default config is usual for static pages. For example I've:

<IfModule mpm_prefork_module>
      StartServers          4
      MinSpareServers       2
      MaxSpareServers       8
      MaxClients           30
      MaxRequestsPerChild  1000
</IfModule>

It seems to work quite well. Also, I had to install mod_bw in apache to
limit the max number of connection from the same ip, to calm down search
engines and bots that download every pdf they find in the world... :-)


Thanks, Yuri, for the suggestions. I have incorporated the above changes
in the apache config file. Keeping a watch on the web server behaviour.
Will let the list know if the changes made to the apache config file is
helping us or not.

Also, I have noticed that the following error message has been reported
quite often in the web server error log file:

-----------------EPrints System Error ----------------------------
Error connecting to MySQL server: Too many connections. To fix this
increase max_connections in my.cnf:

[mysqld]
max_connections=300

------------------------------------------------------------------

Should I incorporate the above 'mysqld' directive or will the changes made
in the apache config take care of the MySQL error?

if they happens again, yes. But I think they're related to many
connections you've.

  This are my my.cnf modifications for tuning:

key_buffer     = 16M
max_allowed_packet   = 16M
thread_stack      = 128K
thread_cache_size = 40
table_cache = 1024

# * Query Cache Configuration<- I don't know if this 2 are already default
#
query_cache_limit       = 1M
query_cache_size        = 16M


you can see how many queries your mysql is processing doing this:

show processlist;

opening mysql client. Then you can see how much loaded your mysql is.




Thanks, Yuri.

Its been almost 24 hours since I made the changes in the apache config
file. What I have noticed after the changes is that the number of httpd
processes is averaging less than 30. Before making the changes in config
file, I have noticed that at times the number of httpd processes would be
130+. That's when the server would freeze.

So, the changes in the config file is keeping the number of httpd process
under control. However, just a while ago, I noticed that the eprints
server was taking a long time to respond. I checked the number of httpd
processes, there were 22 of them and the 'top' command produced the
following output:

top - 17:03:33 up 5 days,  5:17,  1 user,  load average: 2.94, 3.55, 2.80
Tasks: 167 total,   2 running, 165 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.6%us,  0.6%sy,  0.0%ni, 49.2%id, 24.6%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   4051536k total,  4025724k used,    25812k free,     5304k buffers
Swap:  8193108k total,  4098224k used,  4094884k free,    66116k cached

There is clear indication that the  free memory is quite low. I restarted
the httpd server and the output of the 'top' command right after the
restart is:

top - 17:05:57 up 5 days,  5:19,  1 user,  load average: 12.27, 6.11, 3.78
Tasks: 160 total,   1 running, 159 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.2%us,  0.2%sy,  0.0%ni, 92.8%id,  4.8%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   4051536k total,   335168k used,  3716368k free,     7644k buffers
Swap:  8193108k total,   263748k used,  7929360k free,    69500k cached

More than 3.5GB of main memory has been freed after the httpd restart. Is
there anything else that needs to be done to maintain the optimal memory
level to ensure that the eprints server response level is consistent?

Thanks for your attention, Francis




Thanks once again.

    - Francis

JRDTML, IISc
Bengaluru, India

eprints.iisc.ernet.in



Il 19/05/2012 13:22, Francis Jayakanth ha scritto:

Dear Members, Our eprints server freezes (stops responding) almost every
week or so. The only solution to bring up the server again is to do a hard
boot.

I have looked into the Web Server error log file and find that around the
time the system freezes, the following error is recorded in the error log
file:

[Thu May 18 06:10:05 2012] [error] (12)Cannot allocate memory: fork:
Unable to fork new process

So, the obvious reason for the system freeze is, the system is running
out of main memory. Once the server is re-booted, everything is back to
normal, but with the passage of time, the free memory keeps gradually
dropping.

The 'top' command issued right ofter the re-boot, gives the
following output:

top - 12:41:49 up 2 days, 55 min,  1 user,  load average: 0.90, 0.52, 0.35
Tasks: 167 total,   1 running, 166 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us,  0.2%sy,  0.0%ni, 97.3%id,  1.3%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   4051536k total,  1162736k used,  2888800k free,    23300k buffers
Swap:  8193108k total,   378836k used,  7814272k free,   438404k cached

Over the period of few hours, the the usage of the main memory gradually
goes up.

The output of the 'top' command after 3 hours is:

top - 15:42:07 up 2 days,  4:02,  1 user,  load average: 0.23, 0.22, 0.18
Tasks: 173 total,   1 running, 172 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   4051536k total,  3048740k used,  1002796k free,   129020k buffers
Swap:  8193108k total,   378828k used,  7814280k free,   796732k cached

For a time gap of 3 hours, the free memory size has come down from
2888800k to 1002796k. The drop will continue over the period of time and
eventually, the system freezes.

I restarted the apache server after about 3 hours and the output of the
'top' command is:

top - 15:52:59 up 2 days,  4:07,  1 user,  load average: 0.11, 0.14, 0.16
Tasks: 168 total,   1 running, 167 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   4051536k total,  1686412k used,  2365124k free,   130712k buffers
Swap:  8193108k total,   378820k used,  7814288k free,   801620k cached

More than 1.3 GB of memory has been freed after restarting the apache
server.

How do I ensure that memory level doesn't gradually drop and eventually
cause the server to freeze? Should something be done at the apache
configuration level?

Any help/pointers to overcome the said problem will of great help to us.

We have recently upgraded our eprints software from eprints-3.2.8 to
eprints-3.3.9. The above behaviour was very much there with the 3.2.8
version as well.

Apache Server version: Apache/2.2.3 on 64-bit RHES

Many thanks in advance for your attention.


     - Francis

JRD Tata Memorial Library
IISc, Bengaluru, India


*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.