EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #10019

RE: [EP-tech] Server nightmare

To: "eprints-tech@ecs.soton.ac.uk" <eprints-tech@ecs.soton.ac.uk>
Subject: RE: [EP-tech] Server nightmare
From: Will Hughes <w.p.hughes@reading.ac.uk>
Date: Thu, 6 Mar 2025 11:08:52 +0000

CAUTION: This e-mail originated outside the University of Southampton.

Hi Yuri

I have very few records, as I am still in the configuration stage. I have been running with just 9 records, but now I am testing my RIS import process and developing it a little, breaking and repairing things as I go! As to how many views I have configured – only those that came with the pub flavour, I think. In the ..html/en/views folder, there are 298 files in 4 folders – so not a lot, really.

My plan this morning is to run:
sudo -u eprints /opt/eprints3/bin/epadmin erase_eprints arcom
And start again with the small set of test data (4 records). I think I need to clean out all the logs on the server!

The result you ask about:

eprints@ubuntu-s-1vcpu-1gb-lon1-01:~$ df -i /opt/eprints3/

Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/vda1 3010560 137491 2873069 5% /

eprints@ubuntu-s-1vcpu-1gb-lon1-01:~$ df -i /opt/eprints3/archives/arcom

Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/vda1 3010560 137491 2873069 5% /

eprints@ubuntu-s-1vcpu-1gb-lon1-01:~$ df -h

Filesystem Size Used Avail Use% Mounted on

tmpfs 197M 1.1M 196M 1% /run

/dev/vda1 24G 6.5G 17G 28% /

tmpfs 984M 0 984M 0% /dev/shm

tmpfs 5.0M 0 5.0M 0% /run/lock

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-udev-load-credentials.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-sysctl.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-tmpfiles-setup-dev-early.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-tmpfiles-setup-dev.service

/dev/vda13 989M 114M 808M 13% /boot

/dev/vda15 105M 6.1M 99M 6% /boot/efi

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-tmpfiles-setup.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-resolved.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-networkd.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/serial-getty@ttyS0.service

tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service

tmpfs 1.0G 0 1.0G 0% /tmp

tmpfs 197M 12K 197M 1% /run/user/1000

While I left it alone overnight, everything seems to have calmed down. I am about to revert my build to the most recent stable version then continue to experiment with revising the RIS.pm module, and defining new variables that I need based on the data coming in.

For context, I am not running a repository, as such, just a metadata catalogue. So I am not storing any papers in my repository and not having users do any manual inputs. All my data will be coming in through RIS files. I collect the data in EndNote first, so that I can clean it and produce uniform RIS files from there. Currently, I have about 20,000 journal papers 5,000 doctoral theses and 2,000 conference papers (metadata only). The audience for this is an association of researchers in a specific field.

I would welcome your observations on the results above and any suggestions you might have.

Best wishes

Will

From: eprints-tech-request@ecs.soton.ac.uk <eprints-tech-request@ecs.soton.ac.uk> On Behalf Of Yuri
Sent: 06 March 2025 07:39
To: eprints-tech@ecs.soton.ac.uk
Subject: Re: [EP-tech] Server nightmare

CAUTION: This e-mail originated outside the University of Southampton.

Do you have some of this messages?

generate_views can create a lot of files. How many "archive" eprints records do you have? How many views you've configured? what is the result of the command:

df -i /opt/eprints3/

and

df -i /opt/eprints3/archives/ARCHIVEID

and df -h

The first 2 commands should output the filesystem's inode usage, the third the filesystem usage.

Il 05/03/25 23:49, Will Hughes ha scritto:

CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

Hi

In the process of working through my configuration, something has gone severely wrong. When I execute the commend:

sudo -u eprints /opt/eprints3/bin/generate_views ARCHIVEID

The server goes crazy and gets stuck generating thousands of error messages, CPU whacks up to 100%, runs out of memory, and I can barely get it back under control. I have to kill processes, etc. Below are some of the messages from the error log after I stopped systemctl. Is this an Eprints problem or a Server problem?

Best wishes

Will

Will Hughes

Emeritus Professor of Construction Management and Economics

School of the Built Environment

University of Reading, PO Box 219, Whiteknights

Reading, RG6 6DF, UK

Last login: Wed Mar 5 15:41:38 2025 from 162.243.190.66

root@ubuntu-s-1vcpu-1gb-lon1-01:~# cd /var/log/apache2/

root@ubuntu-s-1vcpu-1gb-lon1-01:/var/log/apache2# tail -n 100 error.log

[Wed Mar 05 15:35:55.279733 2025] [perl:error] [pid 384245:tid 384251] [client 165.227.173.50:11611] EPrints::abort()\n

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

DBI connect('database=arcom;host=localhost;mysql_enable_utf8=1','arcom',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 693.

        DBI::__ANON__(undef, undef) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 748

        DBI::connect("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5e2255e08)) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 587

        DBI::connect_cached("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5d89505b0)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 314

        EPrints::Database::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database/mysql.pm line 270

        EPrints::Database::mysql::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 193

        EPrints::Database::new("EPrints::Database", EPrints::Repository=HASH(0x58f5d6798098)) called at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5897

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

DBI connect('database=arcom;host=localhost;mysql_enable_utf8=1','arcom',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 693.

        DBI::__ANON__(undef, undef) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 748

        DBI::connect("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5e2255e08)) called at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 587

        DBI::connect_cached("DBI", "DBI:mysql:database=arcom;host=localhost;mysql_enable_utf8=1", "arcom", "KfjkO3TxnX00aJkd", HASH(0x58f5d89505b0)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 314

        EPrints::Database::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database/mysql.pm line 270

        EPrints::Database::mysql::connect(EPrints::Database::mysql=HASH(0x58f5e2266770)) called at /opt/eprints3/perl_lib/EPrints/Database.pm line 193

        EPrints::Database::new("EPrints::Database", EPrints::Repository=HASH(0x58f5d6798098)) called at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5897

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0
------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

[Wed Mar 05 15:41:38.087893 2025] [perl:error] [pid 384596:tid 384646] [client 204.76.203.15:60318] EPrints::abort()\n

[Wed Mar 05 15:41:38.087986 2025] [perl:error] [pid 384677:tid 384681] [client 43.135.179.102:45808] EPrints::abort()\n

[Wed Mar 05 15:48:12.136728 2025] [mpm_event:notice] [pid 355829:tid 355829] AH00491: caught SIGTERM, shutting down

[Wed Mar 05 15:49:03.664635 2025] [perl:notice] [pid 385274:tid 385274] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.026706 2025] [perl:notice] [pid 385549:tid 385549] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.798344 2025] [perl:notice] [pid 385549:tid 385549] EPrints archives loaded: arcom

[Wed Mar 05 15:56:02.227382 2025] [perl:notice] [pid 385550:tid 385550] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom

[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

[Wed Mar 05 17:43:22.593931 2025] [core:error] [pid 385580:tid 385587] [client 188.93.209.95:47614] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)

[Wed Mar 05 17:43:22.845087 2025] [core:error] [pid 385552:tid 385571] [client 188.93.209.95:47622] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)

[Wed Mar 05 19:28:04.830899 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00492: caught SIGWINCH, shutting down gracefully

[Wed Mar 05 19:28:09.383704 2025] [perl:notice] [pid 390458:tid 390458] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:09.898251 2025] [perl:notice] [pid 390458:tid 390458] EPrints archives loaded: arcom

[Wed Mar 05 19:28:11.232836 2025] [perl:notice] [pid 390459:tid 390459] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:11.734390 2025] [perl:notice] [pid 390459:tid 390459] EPrints archives loaded: arcom

[Wed Mar 05 19:28:12.150157 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 19:28:12.150244 2025] [core:notice] [pid 390459:tid 390459] AH00094: Command line: '/usr/sbin/apache2'

[Wed Mar 05 20:24:32.436453 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00492: caught SIGWINCH, shutting down gracefully

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

------------------------------------------------------------------

----------- EPrints System Error 2025-03-05T15:41:38Z ------------

------------------------------------------------------------------

Error connecting to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

------------------------------------------------------------------

at /opt/eprints3/perl_lib/EPrints/Repository.pm line 5899.

        EPrints::Repository::init_from_request(EPrints::Repository=HASH(0x58f5d6798098), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints.pm line 420

        EPrints::init_from_request(EPrints=HASH(0x58f5d6798038), Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at /opt/eprints3/perl_lib/EPrints/Apache/Rewrite.pm line 70

        EPrints::Apache::Rewrite::handler(Apache2::RequestRec=SCALAR(0x58f5e260a8e8)) called at -e line 0

        eval {...} called at -e line 0

[Wed Mar 05 15:41:38.087893 2025] [perl:error] [pid 384596:tid 384646] [client 204.76.203.15:60318] EPrints::abort()\n

[Wed Mar 05 15:41:38.087986 2025] [perl:error] [pid 384677:tid 384681] [client 43.135.179.102:45808] EPrints::abort()\n

[Wed Mar 05 15:48:12.136728 2025] [mpm_event:notice] [pid 355829:tid 355829] AH00491: caught SIGTERM, shutting down

[Wed Mar 05 15:49:03.664635 2025] [perl:notice] [pid 385274:tid 385274] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.026706 2025] [perl:notice] [pid 385549:tid 385549] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:00.798344 2025] [perl:notice] [pid 385549:tid 385549] EPrints archives loaded: arcom

[Wed Mar 05 15:56:02.227382 2025] [perl:notice] [pid 385550:tid 385550] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 15:56:02.734088 2025] [perl:notice] [pid 385550:tid 385550] EPrints archives loaded: arcom

[Wed Mar 05 15:56:03.176782 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 15:56:03.176856 2025] [core:notice] [pid 385550:tid 385550] AH00094: Command line: '/usr/sbin/apache2'

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

/opt/eprints3/archives/arcom/var/last_changed.timestamp has been modified since the repository config was loaded: reloading!

[Wed Mar 05 17:43:22.593931 2025] [core:error] [pid 385580:tid 385587] [client 188.93.209.95:47614] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)

[Wed Mar 05 17:43:22.845087 2025] [core:error] [pid 385552:tid 385571] [client 188.93.209.95:47622] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)

[Wed Mar 05 19:28:04.830899 2025] [mpm_event:notice] [pid 385550:tid 385550] AH00492: caught SIGWINCH, shutting down gracefully

[Wed Mar 05 19:28:09.383704 2025] [perl:notice] [pid 390458:tid 390458] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:09.898251 2025] [perl:notice] [pid 390458:tid 390458] EPrints archives loaded: arcom

[Wed Mar 05 19:28:11.232836 2025] [perl:notice] [pid 390459:tid 390459] Warning! Running EPrints under threads is experimental and liable to break

[Wed Mar 05 19:28:11.734390 2025] [perl:notice] [pid 390459:tid 390459] EPrints archives loaded: arcom

[Wed Mar 05 19:28:12.150157 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00489: Apache/2.4.62 (Ubuntu) mod_perl/2.0.13 Perl/v5.38.2 configured -- resuming normal operations

[Wed Mar 05 19:28:12.150244 2025] [core:notice] [pid 390459:tid 390459] AH00094: Command line: '/usr/sbin/apache2'

[Wed Mar 05 20:24:32.436453 2025] [mpm_event:notice] [pid 390459:tid 390459] AH00492: caught SIGWINCH, shutting down gracefully
*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/
 

References:
- [EP-tech] Server nightmare
  - From: Will Hughes <w.p.hughes@reading.ac.uk>
- Re: [EP-tech] Server nightmare
  - From: Yuri <yurj@alfa.it>

Prev by Date: Re: [EP-tech] Server nightmare
Next by Date: [EP-tech] Tracing errors
Previous by thread: Re: [EP-tech] Server nightmare
Next by thread: Re: [EP-tech] Server nightmare
Index(es):
- Date
- Thread