EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #10016

Re: [EP-tech] Server nightmare

To: eprints-tech@ecs.soton.ac.uk
Subject: Re: [EP-tech] Server nightmare
From: Yuri <yurj@alfa.it>
Date: Thu, 6 Mar 2025 08:38:56 +0100

CAUTION: This e-mail originated outside the University of Southampton.

Do you have some of this messages?

generate_views can create a lot of files. How many "archive" eprints records do you have? How many views you've configured? what is the result of the command:

df -i /opt/eprints3/

and

df -i /opt/eprints3/archives/ARCHIVEID

and df -h

The first 2 commands should output the filesystem's inode usage, the third the filesystem usage.

Il 05/03/25 23:49, Will Hughes ha scritto:

CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

Hi

In the process of working through my configuration, something has gone severely wrong. When I execute the commend:

sudo -u eprints /opt/eprints3/bin/generate_views ARCHIVEID

The server goes crazy and gets stuck generating thousands of error messages, CPU whacks up to 100%, runs out of memory, and I can barely get it back under control. I have to kill processes, etc. Below are some of the messages from the error log after I stopped systemctl. Is this an Eprints problem or a Server problem?

Best wishes

Will

Will Hughes

Emeritus Professor of Construction Management and Economics

School of the Built Environment

University of Reading, PO Box 219, Whiteknights

Reading, RG6 6DF, UK

Last login: Wed Mar 5 15:41:38 2025 from 162.243.190.66

root@ubuntu-s-1vcpu-1gb-lon1-01:~# cd /var/log/apache2/

root@ubuntu-s-1vcpu-1gb-lon1-01:/var/log/apache2# tail -n 100 error.log

[Wed Mar 05 15:35:55.279733 2025] [perl:error] [pid 384245:tid 384251] [client 165.227.173.50:11611] EPrints::abort()\n

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

DBI connect('database=arcom;host=localhost;mysql_enable_utf8=1','arcom',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 693.

        DBI::__ANON__(undef, undef) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 748

        DBI::connect("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5e2255e08)) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 587

        DBI::connect_cached("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5d89505b0)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 314

        EPrints::Database::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database/mysql.pm line 270

        EPrints::Database::mysql::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 193

        EPrints::Database::new("EPrints::Database", EPrints::Repository=HASH(0x58f5d6798098)) called at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5897

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

DBI connect('database=arcom;host=localhost;mysql_enable_utf8=1','arcom',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 693.

        DBI::__ANON__(undef, undef) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 748

        DBI::connect("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5e2255e08)) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 587

        DBI::connect_cached("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5d89505b0)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 314

        EPrints::Database::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database/mysql.pm line 270

        EPrints::Database::mysql::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 193

        EPrints::Database::new("EPrints::Database", EPrints::Repository=HASH(0x58f5d6798098)) called at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5897

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0
------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

[Wed Mar 05 15:41:38.087893 2025] [perl:error] [pid 384596:tid 384646] [client 204.76.203.15:60318] EPrints::abort()\n

[Wed Mar 05 15:41:38.087986 2025] [perl:error] [pid 384677:tid 384681] [client 43.135.179.102:45808] EPrints::abort()\n

[Wed Mar 05 15:48:12.136728 2025] [mpm_event:notice] [pid 355829:tid 355829] AH00491: caught SIGTERM, shutting down

[Wed Mar 05 15:49:03.664635 2025] [perl:notice] [pid 385274:tid 385274] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.026706 2025] [perl:notice] [pid 385549:tid 385549] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.798344 2025] [perl:notice] [pid 385549:tid 385549] EPrints archives loaded: arcom

[Wed Mar 05 15:56:02.227382 2025] [perl:notice] [pid 385550:tid 385550] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom

[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

[Wed Mar 05 17:43:22.593931 2025] [core:error] [pid 385580:tid 385587] [client 188.93.209.95:47614] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)

[Wed Mar 05 17:43:22.845087 2025] [core:error] [pid 385552:tid 385571] [client 188.93.209.95:47622] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)

[Wed Mar 05 19:28:04.830899 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00492: caught SIGWINCH, shutting down gracefully

[Wed Mar 05 19:28:09.383704 2025] [perl:notice] [pid 390458:tid 390458] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:09.898251 2025] [perl:notice] [pid 390458:tid 390458] EPrints archives loaded: arcom

[Wed Mar 05 19:28:11.232836 2025] [perl:notice] [pid 390459:tid 390459] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:11.734390 2025] [perl:notice] [pid 390459:tid 390459] EPrints archives loaded: arcom

[Wed Mar 05 19:28:12.150157 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 19:28:12.150244 2025] [core:notice] [pid 390459:tid 390459] AH00094: Command line: '/usr/sbin/apache2'

[Wed Mar 05 20:24:32.436453 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00492: caught SIGWINCH, shutting down gracefully

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

[Wed Mar 05 15:41:38.087893 2025] [perl:error] [pid 384596:tid 384646] [client 204.76.203.15:60318] EPrints::abort()\n

[Wed Mar 05 15:41:38.087986 2025] [perl:error] [pid 384677:tid 384681] [client 43.135.179.102:45808] EPrints::abort()\n

[Wed Mar 05 15:48:12.136728 2025] [mpm_event:notice] [pid 355829:tid 355829] AH00491: caught SIGTERM, shutting down

[Wed Mar 05 15:49:03.664635 2025] [perl:notice] [pid 385274:tid 385274] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.026706 2025] [perl:notice] [pid 385549:tid 385549] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.798344 2025] [perl:notice] [pid 385549:tid 385549] EPrints archives loaded: arcom

[Wed Mar 05 15:56:02.227382 2025] [perl:notice] [pid 385550:tid 385550] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom

[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

[Wed Mar 05 17:43:22.593931 2025] [core:error] [pid 385580:tid 385587] [client 188.93.209.95:47614] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)

[Wed Mar 05 17:43:22.845087 2025] [core:error] [pid 385552:tid 385571] [client 188.93.209.95:47622] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)

[Wed Mar 05 19:28:04.830899 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00492: caught SIGWINCH, shutting down gracefully

[Wed Mar 05 19:28:09.383704 2025] [perl:notice] [pid 390458:tid 390458] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:09.898251 2025] [perl:notice] [pid 390458:tid 390458] EPrints archives loaded: arcom

[Wed Mar 05 19:28:11.232836 2025] [perl:notice] [pid 390459:tid 390459] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:11.734390 2025] [perl:notice] [pid 390459:tid 390459] EPrints archives loaded: arcom

[Wed Mar 05 19:28:12.150157 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 19:28:12.150244 2025] [core:notice] [pid 390459:tid 390459] AH00094: Command line: '/usr/sbin/apache2'

[Wed Mar 05 20:24:32.436453 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00492: caught SIGWINCH, shutting down gracefully
*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/

Follow-Ups:
- RE: [EP-tech] Server nightmare
  - From: Will Hughes <w.p.hughes@reading.ac.uk>

References:
- [EP-tech] Server nightmare
  - From: Will Hughes <w.p.hughes@reading.ac.uk>

Prev by Date: [EP-tech] Server nightmare
Next by Date: Re: [EP-tech] Server nightmare
Previous by thread: [EP-tech] Server nightmare
Next by thread: RE: [EP-tech] Server nightmare
Index(es):
- Date
- Thread