EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #10017

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Server nightmare

Hi Will,

The main error message I see there is that the database is probably not running and that is why it cannot connect.  That sounds like a likely symptom of generate_views using up a lot of memory and CPU and gumming everything up.  Are you actually seeing any error messages directly from running generate_views rather than what you are finding in the Apache logs.

generate_views can be quite processor-intensive, especially if you have added extra browse views.  If you have limited resources (i.e memory, CPU processing), etc.) on your server, it is probably worth running this command with nice.  E.g.

sudo -u eprints  nice -n 10 /opt/eprints3/bin/generate_views ARCHIVEID

You can increase the nice value up to 19 to further lower the priority and hopefully use less resources.  There may be a particular browse view you have configured that especially processor-intensive.  You could try generating each one in turn to see if one particular one makes it unhappy:

sudo -u eprints  /opt/eprints3/bin/generate_views ARCHIVEID  --view VIEWID

Similarly, if you run the command in verbose mode and you may better see where it is getting overwhelmed:

sudo -u eprints  /opt/eprints3/bin/generate_views ARCHIVEID  --verbose

Technically, you do not need to run generate_views at all as browse view pages are cached for 24 hours and then the next request after that regenerates them on the fly.  However, this can mean the user has to wait a while for browse view pages to load.  A half-way house is to regenerate the menu pages that for example list all the authors with links off to their pages that actually list all their publications:

sudo -u eprints  /opt/eprints3/bin/generate_views ARCHIVEID --generate menus

This is something I do for quite a few repositories, as they may have thousands of authors or generating the listing pages for listing pages can be very slow for the year browse views as they have thousands of items.  Obviously, these will still be slower to load for the first user to request them after the cache has expired.  However, this is often more tolerable for a user, as they are only waiting for their final destination to load rather than a page they then need to click on again (i.e. a browse view menu page) before the get what they actually want.

As an addendum: EPrints 3.4.6 has hopefully fixed the issues with running Apache in MPM event mode.  This means the "Warning! Running EPrints under threads is experimental and liable to break" can probably now be safely ignored.  However, it was left in for now, in case subsequent issues were found.  Hopefully, this warning message can be removed for the next version of EPrints.

Some of the other log messages you show are just Apache reloading the EPrints configuration or as a result of someone clicking the "Reload Configuration" button in the admin web interface:

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom
[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations
[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

Regards

David Newman



On 05/03/2025 10:49 pm, Will Hughes wrote:
CAUTION: This e-mail originated outside the University of Southampton.
CAUTION: This e-mail originated outside the University of Southampton.

Hi

 

In the process of working through my configuration, something has gone severely wrong. When I execute the commend:

 

sudo -u eprints /opt/eprints3/bin/generate_views ARCHIVEID

 

The server goes crazy and gets stuck generating thousands of error messages, CPU whacks up to 100%, runs out of memory, and I can barely get it back under control. I have to kill processes, etc. Below are some of the messages from the error log after I stopped systemctl. Is this an Eprints problem or a Server problem?

 

Best wishes

 

Will   

 

Will Hughes

Emeritus Professor of Construction Management and Economics

School of the Built Environment     

University of Reading, PO Box 219, Whiteknights

Reading, RG6 6DF, UK

 

 

 

Last login: Wed Mar  5 15:41:38 2025 from 162.243.190.66

root@ubuntu-s-1vcpu-1gb-lon1-01:~# cd /var/log/apache2/

root@ubuntu-s-1vcpu-1gb-lon1-01:/var/log/apache2# tail -n 100 error.log

[Wed Mar 05 15:35:55.279733 2025] [perl:error] [pid 384245:tid 384251] [client 165.227.173.50:11611] EPrints::abort()\n

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

DBI connect('database=arcom;host=localhost;mysql_enable_utf8=1','arcom',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 693.

        DBI::__ANON__(undef, undef) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 748

        DBI::connect("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5e2255e08)) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 587

        DBI::connect_cached("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5d89505b0)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 314

        EPrints::Database::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database/mysql.pm line 270

        EPrints::Database::mysql::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 193

        EPrints::Database::new("EPrints::Database", EPrints::Repository=HASH(0x58f5d6798098)) called at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5897

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

DBI connect('database=arcom;host=localhost;mysql_enable_utf8=1','arcom',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 693.

        DBI::__ANON__(undef, undef) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 748

        DBI::connect("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5e2255e08)) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 587

        DBI::connect_cached("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5d89505b0)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 314

        EPrints::Database::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database/mysql.pm line 270

        EPrints::Database::mysql::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 193

        EPrints::Database::new("EPrints::Database", EPrints::Repository=HASH(0x58f5d6798098)) called at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5897

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0
------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

[Wed Mar 05 15:41:38.087893 2025] [perl:error] [pid 384596:tid 384646] [client 204.76.203.15:60318] EPrints::abort()\n

[Wed Mar 05 15:41:38.087986 2025] [perl:error] [pid 384677:tid 384681] [client 43.135.179.102:45808] EPrints::abort()\n

[Wed Mar 05 15:48:12.136728 2025] [mpm_event:notice] [pid 355829:tid 355829] AH00491: caught SIGTERM, shutting down

[Wed Mar 05 15:49:03.664635 2025] [perl:notice] [pid 385274:tid 385274] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.026706 2025] [perl:notice] [pid 385549:tid 385549] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.798344 2025] [perl:notice] [pid 385549:tid 385549] EPrints archives loaded: arcom

[Wed Mar 05 15:56:02.227382 2025] [perl:notice] [pid 385550:tid 385550] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom

[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

[Wed Mar 05 17:43:22.593931 2025] [core:error] [pid 385580:tid 385587] [client 188.93.209.95:47614] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)

[Wed Mar 05 17:43:22.845087 2025] [core:error] [pid 385552:tid 385571] [client 188.93.209.95:47622] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)

[Wed Mar 05 19:28:04.830899 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00492: caught SIGWINCH, shutting down gracefully

[Wed Mar 05 19:28:09.383704 2025] [perl:notice] [pid 390458:tid 390458] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:09.898251 2025] [perl:notice] [pid 390458:tid 390458] EPrints archives loaded: arcom

[Wed Mar 05 19:28:11.232836 2025] [perl:notice] [pid 390459:tid 390459] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:11.734390 2025] [perl:notice] [pid 390459:tid 390459] EPrints archives loaded: arcom

[Wed Mar 05 19:28:12.150157 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 19:28:12.150244 2025] [core:notice] [pid 390459:tid 390459] AH00094: Command line: '/usr/sbin/apache2'

[Wed Mar 05 20:24:32.436453 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00492: caught SIGWINCH, shutting down gracefully

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

[Wed Mar 05 15:41:38.087893 2025] [perl:error] [pid 384596:tid 384646] [client 204.76.203.15:60318] EPrints::abort()\n

[Wed Mar 05 15:41:38.087986 2025] [perl:error] [pid 384677:tid 384681] [client 43.135.179.102:45808] EPrints::abort()\n

[Wed Mar 05 15:48:12.136728 2025] [mpm_event:notice] [pid 355829:tid 355829] AH00491: caught SIGTERM, shutting down

[Wed Mar 05 15:49:03.664635 2025] [perl:notice] [pid 385274:tid 385274] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.026706 2025] [perl:notice] [pid 385549:tid 385549] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.798344 2025] [perl:notice] [pid 385549:tid 385549] EPrints archives loaded: arcom

[Wed Mar 05 15:56:02.227382 2025] [perl:notice] [pid 385550:tid 385550] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom

[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

[Wed Mar 05 17:43:22.593931 2025] [core:error] [pid 385580:tid 385587] [client 188.93.209.95:47614] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)

[Wed Mar 05 17:43:22.845087 2025] [core:error] [pid 385552:tid 385571] [client 188.93.209.95:47622] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)

[Wed Mar 05 19:28:04.830899 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00492: caught SIGWINCH, shutting down gracefully

[Wed Mar 05 19:28:09.383704 2025] [perl:notice] [pid 390458:tid 390458] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:09.898251 2025] [perl:notice] [pid 390458:tid 390458] EPrints archives loaded: arcom

[Wed Mar 05 19:28:11.232836 2025] [perl:notice] [pid 390459:tid 390459] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:11.734390 2025] [perl:notice] [pid 390459:tid 390459] EPrints archives loaded: arcom

[Wed Mar 05 19:28:12.150157 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 19:28:12.150244 2025] [core:notice] [pid 390459:tid 390459] AH00094: Command line: '/usr/sbin/apache2'

[Wed Mar 05 20:24:32.436453 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00492: caught SIGWINCH, shutting down gracefully


*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/