EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #09952

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Request for Assistance: Slow Turnaround Time for EPrints Repository

Hi Francis.
I use the following Apache "performance tuning" on a system with roughly half the resources RAM / CPUs but I do have more swap and each CPU is more powerful: 

<IfModule mpm_prefork_module>
  StartServers            2
  MinSpareServers         2
  MaxSpareServers         5
  ServerLimit             30
  MaxRequestWorkers       30
  MaxConnectionsPerChild  1000
</IfModule>
Based on [1] MaxRequestWorkers and MaxClients are synonymous at least for prefork.  Also for prefork (i.e. non-threaded) ServerLimit and MaxRequestWorkers should be the same as if the latter is greater than the former then this will make no difference (you don't really need to set ServerLimit at all its default is very high but always limited by either the default or custom value for MaxRequestWorkers).  The default value for MaxClients / MaxRequestWorkers is 256, so taking this down to 10 is a massive drop.  Some EPrints requests are quite processor intensive, so reducing from the default is sensible.  30 I have found as a sweet spot, where generally requests don't have too wait too long in the queue (as there are 30 workers who can service requests)  but the system resources do not get overwhelmed.  With more RAM and more CPUs, you may be able to set this higher than 30.
Recently, I upped the MaxConnectionsPerChild from 100 to 1000, as I realised the overhead and time of respawning child processes was potentially having an impact on response times [2].  The default value is 0 which means never kill and respawn processes.  Never respawning can sometimes cause the server to run out of memory, typically due to leaks.  Setting to 1000 should not cause you to run out of memory but should improve response times as a request is not having to wait for a process to respawn so frequently.
Beyond your Apache configuration, the version of MariaDB you are running is quite old.  A while ago, I found a performance improvement when I upgraded CentOS 7 servers to use MariaDB 10.5 by installing this from MariaDB's own YUM repository.  However, I don't know if CentOS 7 is still supported by MariaDB's YUM repository, as it went out of support the end of June 2024.  So the more sensible approach may be to build a new server running Red Hat Enterprise Linux 9 or one of its derivatives and then you will get MariaDB 10.5 by default.
Above is just general advice.  Your repository's specific traffic profile may mean that different settings work better for you.  If a lot of requests are for static or pre-cached pages, then the MaxRequestWorkers and MaxConnectionsPerChild can possibly be increased further.  If you have a lot of search and/or exportview requests, then lower values may be better.  If you have extensive browse views, it may be best to make sure both the menus and listing pages are cached by generate_views at quieter times of day rather than letting them be generated on the next request are the cache expires.  Sometimes you can have a bot making hundreds of browse view requests just after the cache expires, which can really put a high load on your server all of a sudden. 

Regards

David Newman

[1] https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.4%2Fmod%2Fmpm_common.html%23maxrequestworkers&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6f0604c808f449a282d08dd3df9648f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638734863514136846%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Y7KFIpd7Q2gmGdj4E%2F3xehZFPiRGkvmhS5slpdSBHIc%3D&reserved=0
[2] https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.4%2Fmod%2Fmpm_common.html%23maxconnectionsperchild&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6f0604c808f449a282d08dd3df9648f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638734863514159266%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=4XLDQoeSPe38evdRwm6yQCRb10HQP5NXo%2FkENiOebmM%3D&reserved=0 

On 25/01/2025 3:59 pm, Francis Jayakanth wrote:

CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

Dear EPrints Technical Support, We are currently experiencing
significantly very slow turnaround times on our EPrints repository at
eprints.iisc.ac.in, and we would appreciate any assistance or
suggestions to improve the performance.

System Configuration:

OS: CentOS 7
EPrints Version: 3.4.1-rc2
Web Server: Apache httpd 2.4.6
Database: MariaDB (mysql Ver 15.1 Distrib 5.5.68-MariaDB)

Apache Configuration:

We have configured the Apache web server using the following settings:

apache
Copy
<IfModule prefork.c>
     StartServers       4
     MinSpareServers    4
     MaxSpareServers    8
     ServerLimit        10
     MaxClients         10
     MaxRequestsPerChild 50
</IfModule>

System Resources:

RAM:
Mem:           7.8G        5.4G        1.7G        7.6M        663M        2.1G
Swap:          2.0G        1.7G        284M

CPU:
4 CPUs (Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz)

Observations:

- Process turnaround times (uploading, reviewing, browsing, logging
in, searching, etc.) are much slower than expected. The performance is
very poor despite restarting the web server often. Every time the web
server is restarted, there is a noticeable improvement in the
turnaround time, but the performance degrades quickly.

We would appreciate any suggestions on what we could do in our current
setup or any recommendations for troubleshooting the performance
bottleneck.

Looking forward to your guidance.

Thanking you,

Best regards, Francis

*** Options: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.eprints.org%2Fw%2FEprints-tech_Mailing_List&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6f0604c808f449a282d08dd3df9648f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638734863514176787%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=npmCcbMKqCE9RuX8zMYbMBGLHzJm89Qs7Pu%2BV2FyULU%3D&reserved=0
*** Archive: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6f0604c808f449a282d08dd3df9648f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638734863514192988%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GLGOvRMEGjEGLR3QHzPRwpGTiTP5DXIuuM0Plgeic9s%3D&reserved=0
*** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.eprints.org%2F&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7Cc6f0604c808f449a282d08dd3df9648f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638734863514208828%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AwCvRnrJva84YFbvBHNIjCcZlGF4M8%2Bg%2FRTNexN1JEs%3D&reserved=0