EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #09957

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Request for Assistance: Slow Turnaround Time for EPrints Repository

  • To: David R Newman <drn@ecs.soton.ac.uk>
  • Subject: Re: [EP-tech] Request for Assistance: Slow Turnaround Time for EPrints Repository
  • From: Francis Jayakanth <fjayakanth@gmail.com>
  • Date: Mon, 27 Jan 2025 22:18:37 +0530
CAUTION: This e-mail originated outside the University of Southampton.

Hi David, Thank you for your detailed reply and the helpful
suggestions to improve the performance of our EPrints repository. I
have implemented the changes you recommended in the Apache
"performance tuning" settings and noticed a marginal improvement in
the system’s performance.

Additionally, we are planning to migrate the repository to the RHEL
platform. We hope this move will address the performance issues and
provide a more stable environment.

Once again, I appreciate your insights and will keep you updated on
our progress.

Best regards, Francis

On Sun, Jan 26, 2025 at 4:35 PM David R Newman <drn@ecs.soton.ac.uk> wrote:
>
> Hi Francis.
>
> I use the following Apache "performance tuning" on a system with roughly
> half the resources RAM / CPUs but I do have more swap and each CPU is
> more powerful:
>
> <IfModule mpm_prefork_module>
>    StartServers            2
>    MinSpareServers         2
>    MaxSpareServers         5
>    ServerLimit             30
>    MaxRequestWorkers       30
>    MaxConnectionsPerChild  1000
> </IfModule>
>
> Based on [1] MaxRequestWorkers and MaxClients are synonymous at least
> for prefork.  Also for prefork (i.e. non-threaded) ServerLimit and
> MaxRequestWorkers should be the same as if the latter is greater than
> the former then this will make no difference (you don't really need to
> set ServerLimit at all its default is very high but always limited by
> either the default or custom value for MaxRequestWorkers).  The default
> value for MaxClients / MaxRequestWorkers is 256, so taking this down to
> 10 is a massive drop.  Some EPrints requests are quite processor
> intensive, so reducing from the default is sensible.  30 I have found as
> a sweet spot, where generally requests don't have too wait too long in
> the queue (as there are 30 workers who can service requests)  but the
> system resources do not get overwhelmed.  With more RAM and more CPUs,
> you may be able to set this higher than 30.
>
> Recently, I upped the MaxConnectionsPerChild from 100 to 1000, as I
> realised the overhead and time of respawning child processes was
> potentially having an impact on response times [2].  The default value
> is 0 which means never kill and respawn processes.  Never respawning can
> sometimes cause the server to run out of memory, typically due to
> leaks.  Setting to 1000 should not cause you to run out of memory but
> should improve response times as a request is not having to wait for a
> process to respawn so frequently.
>
> Beyond your Apache configuration, the version of MariaDB you are running
> is quite old.  A while ago, I found a performance improvement when I
> upgraded CentOS 7 servers to use MariaDB 10.5 by installing this from
> MariaDB's own YUM repository.  However, I don't know if CentOS 7 is
> still supported by MariaDB's YUM repository, as it went out of support
> the end of June 2024.  So the more sensible approach may be to build a
> new server running Red Hat Enterprise Linux 9 or one of its derivatives
> and then you will get MariaDB 10.5 by default.
>
> Above is just general advice.  Your repository's specific traffic
> profile may mean that different settings work better for you.  If a lot
> of requests are for static or pre-cached pages, then the
> MaxRequestWorkers and MaxConnectionsPerChild can possibly be increased
> further.  If you have a lot of search and/or exportview requests, then
> lower values may be better.  If you have extensive browse views, it may
> be best to make sure both the menus and listing pages are cached by
> generate_views at quieter times of day rather than letting them be
> generated on the next request are the cache expires.  Sometimes you can
> have a bot making hundreds of browse view requests just after the cache
> expires, which can really put a high load on your server all of a sudden.
>
> Regards
>
> David Newman
>
> [1] https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.4%2Fmod%2Fmpm_common.html%23maxrequestworkers&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7C234d1d968f6a4baeb09208dd3ef27a56%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638735934079701919%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=T771zvKhIWUmT1VNIoNjCN6%2FQWQm9bjPhuOogJCw5f0%3D&reserved=0
> [2]
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.4%2Fmod%2Fmpm_common.html%23maxconnectionsperchild&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7C234d1d968f6a4baeb09208dd3ef27a56%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638735934079722508%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=xuQvWHKp9uBDKupEesB2w3sidobG%2BSHp0M5wZ48TZP8%3D&reserved=0
>
> On 25/01/2025 3:59 pm, Francis Jayakanth wrote:
> > CAUTION: This e-mail originated outside the University of Southampton.
> >
> > CAUTION: This e-mail originated outside the University of Southampton.
> >
> > Dear EPrints Technical Support, We are currently experiencing
> > significantly very slow turnaround times on our EPrints repository at
> > eprints.iisc.ac.in, and we would appreciate any assistance or
> > suggestions to improve the performance.
> >
> > System Configuration:
> >
> > OS: CentOS 7
> > EPrints Version: 3.4.1-rc2
> > Web Server: Apache httpd 2.4.6
> > Database: MariaDB (mysql Ver 15.1 Distrib 5.5.68-MariaDB)
> >
> > Apache Configuration:
> >
> > We have configured the Apache web server using the following settings:
> >
> > apache
> > Copy
> > <IfModule prefork.c>
> >      StartServers       4
> >      MinSpareServers    4
> >      MaxSpareServers    8
> >      ServerLimit        10
> >      MaxClients         10
> >      MaxRequestsPerChild 50
> > </IfModule>
> >
> > System Resources:
> >
> > RAM:
> > Mem:           7.8G        5.4G        1.7G        7.6M        663M        2.1G
> > Swap:          2.0G        1.7G        284M
> >
> > CPU:
> > 4 CPUs (Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz)
> >
> > Observations:
> >
> > - Process turnaround times (uploading, reviewing, browsing, logging
> > in, searching, etc.) are much slower than expected. The performance is
> > very poor despite restarting the web server often. Every time the web
> > server is restarted, there is a noticeable improvement in the
> > turnaround time, but the performance degrades quickly.
> >
> > We would appreciate any suggestions on what we could do in our current
> > setup or any recommendations for troubleshooting the performance
> > bottleneck.
> >
> > Looking forward to your guidance.
> >
> > Thanking you,
> >
> > Best regards, Francis
> >
> > *** Options: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.eprints.org%2Fw%2FEprints-tech_Mailing_List&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7C234d1d968f6a4baeb09208dd3ef27a56%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638735934079735976%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=qRdM7OkvyaqgfiwGXIlIN3Ffr6eaS6U%2BJMsKLCwqEI4%3D&reserved=0
> > *** Archive: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7C234d1d968f6a4baeb09208dd3ef27a56%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638735934079748474%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=2Wxaxxe0RMZIocMYvO6piOpRlycbc2PB8aavx6HDx%2Bg%3D&reserved=0
> > *** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.eprints.org%2F&data=05%7C02%7Ceprints-tech%40ecs.soton.ac.uk%7C234d1d968f6a4baeb09208dd3ef27a56%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638735934079760331%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=wNcJPBOtu87tE2bZph0DTeb5NmvCepQrkq6TF8aEjqE%3D&reserved=0
> >
>