CAUTION: This e-mail originated outside the University of Southampton.

Hi David

Many thanks for your support.

Thanks

Maher

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Tuesday, January 7, 2025 1:18 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

Hi Maher,

I don't think there is an further advice I can provide beyond using a memory monitoring tool like Valgrind [1] to analyse the issue.  This looks to be a low-level memory issue that I cannot replicate on a similar operating system.

I think any Valgrind command would need to set flags like:

--track-origins=yes 

--keep-stacktraces=alloc-and-free

This will help track down where memory is not correctly being allocated and freed by an Apache module as this will almost certainly be an issue with the compiled C++ of a module not something EPrints itself does.  I don't know whether the output will give you an idea for a solution, as the logging may not mean much to you unless you have a C/C++ or similar low-level programming background.

Regards

David Newman

[1] https://valgrind.org/

CAUTION: This e-mail originated outside the University of Southampton.

Hi David

 

We changed to mpm_event_module to see if it will help in resolving the error which it did not.  And yes switching to mpm_event will load the cgid_module instead of cgi.

 

Thanks

Maher

 

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Tuesday, January 7, 2025 12:57 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

 

Hi Maher, It did a comparison with my httpd modules on EL 9. 5 and I can see you have mpm_event_module loaded (see below) whereas I have mpm_prefork_module loaded. The only other difference is you have cgid_module loaded whereas I have cgi_module. 

ZjQcmQRYFpfptBannerStart

Warning: This email has been sent from outside KFUPM. Do not open links or attachments unless you are sure they are safe.

ZjQcmQRYFpfptBannerEnd

Hi Maher,

It did a comparison with my httpd modules on EL 9.5 and I can see you have mpm_event_module loaded (see below) whereas I have mpm_prefork_module loaded.  The only other difference is you have cgid_module loaded whereas I have cgi_module.  I think this other difference is only because cgid_module loads when you are using mpm_event_module as it is needed when Apache is running multi-threaded.

Regards

David Newman

On 07/01/2025 07:24, Maher Abdellatif Ahmad Qahwash wrote:

CAUTION: This e-mail originated outside the University of Southampton.

Hi David

 

We have uninstalled the CPAN mod_perl and installed it using DNF but still the same error is appearing.  Any other thoughts?

 

Following is the list of modules being loaded:

 

Loaded Modules:

core_module (static)

so_module (static)

http_module (static)

access_compat_module (shared)

actions_module (shared)

alias_module (shared)

allowmethods_module (shared)

auth_basic_module (shared)

auth_digest_module (shared)

authn_anon_module (shared)

authn_core_module (shared)

authn_dbd_module (shared)

authn_dbm_module (shared)

authn_file_module (shared)

authn_socache_module (shared)

authz_core_module (shared)

authz_dbd_module (shared)

authz_dbm_module (shared)

authz_groupfile_module (shared)

authz_host_module (shared)

authz_owner_module (shared)

authz_user_module (shared)

autoindex_module (shared)

cache_module (shared)

cache_disk_module (shared)

cache_socache_module (shared)

data_module (shared)

dbd_module (shared)

deflate_module (shared)

dir_module (shared)

dumpio_module (shared)

echo_module (shared)

env_module (shared)

expires_module (shared)

ext_filter_module (shared)

filter_module (shared)

headers_module (shared)

include_module (shared)

info_module (shared)

log_config_module (shared)

logio_module (shared)

macro_module (shared)

mime_magic_module (shared)

mime_module (shared)

negotiation_module (shared)

remoteip_module (shared)

reqtimeout_module (shared)

request_module (shared)

rewrite_module (shared)

setenvif_module (shared)

slotmem_plain_module (shared)

slotmem_shm_module (shared)

socache_dbm_module (shared)

socache_memcache_module (shared)

socache_redis_module (shared)

socache_shmcb_module (shared)

status_module (shared)

substitute_module (shared)

suexec_module (shared)

unique_id_module (shared)

unixd_module (shared)

userdir_module (shared)

version_module (shared)

vhost_alias_module (shared)

watchdog_module (shared)

brotli_module (shared)

dav_module (shared)

dav_fs_module (shared)

dav_lock_module (shared)

lua_module (shared)

mpm_event_module (shared)

proxy_module (shared)

lbmethod_bybusyness_module (shared)

lbmethod_byrequests_module (shared)

lbmethod_bytraffic_module (shared)

lbmethod_heartbeat_module (shared)

proxy_ajp_module (shared)

proxy_balancer_module (shared)

proxy_connect_module (shared)

proxy_express_module (shared)

proxy_fcgi_module (shared)

proxy_fdpass_module (shared)

proxy_ftp_module (shared)

proxy_http_module (shared)

proxy_hcheck_module (shared)

proxy_scgi_module (shared)

proxy_uwsgi_module (shared)

proxy_wstunnel_module (shared)

ssl_module (shared)

systemd_module (shared)

cgid_module (shared)

perl_module (shared)

http2_module (shared)

proxy_http2_module (shared)

 

 

Thanks

Maher

 

 

 

 

 

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Monday, January 6, 2025 2:52 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

 

Hi Maher, We use EPEL, which can usually be installed by running: dnf install epel-release Regards David Newman On 06/01/2025 11: 49, Maher Abdellatif Ahmad Qahwash wrote: CAUTION: This e-mail originated outside the University of Southampton. 

ZjQcmQRYFpfptBannerStart

Warning: This email has been sent from outside KFUPM. Do not open links or attachments unless you are sure they are safe.

ZjQcmQRYFpfptBannerEnd

Hi Maher,

We use EPEL, which can usually be installed by running:

dnf install epel-release

Regards

David Newman

On 06/01/2025 11:49, Maher Abdellatif Ahmad Qahwash wrote:

CAUTION: This e-mail originated outside the University of Southampton.

Hi David

 

The problem is mod_perl not available using dnf

[root@eprints2 ~]# dnf install mod_perl

Updating Subscription Management repositories.

Last metadata expiration check: 2:57:30 ago on Mon 06 Jan 2025 11:51:35 AM +03.

No match for argument: mod_perl

Error: Unable to find a match: mod_perl

 

Is there any repository we need to fisrt add which contains the mod_perl?

 

Thanks

Maher

 

 

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Monday, January 6, 2025 2:46 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

 

Hi Maher, There is higher chance that if you install mod_perl2 through CPAN rather than installing the mod_perl provided by an OS package it will be more buggy (as it probably won't have been as thoroughly tested as the version adopted by RHEL

ZjQcmQRYFpfptBannerStart

Warning: This email has been sent from outside KFUPM. Do not open links or attachments unless you are sure they are safe.

ZjQcmQRYFpfptBannerEnd

Hi Maher,

There is higher chance that if you install mod_perl2 through CPAN rather than installing the mod_perl provided by an OS package it will be more buggy (as it probably won't have been as thoroughly tested as the version adopted by RHEL 9) or potentially incompatible with the version of Apache RHEL 9 installs.  I would certainly try removing the CPAN version of mod_perl2 and installing mod_perl using dnf/yum.

Regards

David Newman

On 06/01/2025 11:39, Maher Abdellatif Ahmad Qahwash wrote:

CAUTION: This e-mail originated outside the University of Southampton.

Hi David

 

The modules loaded are identical.  As for mod_perl, running the command rpm -qa | grep mod_perl does not give a result.  Since mod_perl is not available using dnf we installed mod_perl2 using cpan.  Could this be the issue?

 

Thanks

Maher

 

 

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Monday, January 6, 2025 2:27 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

 

Hi Maher, Having a test server will be useful but the fact that your test server (running the same version of RHEL and EPrints) has no problem but your production server does suggest an underlying issue rather than an issue with EPrints itself. 

ZjQcmQRYFpfptBannerStart

Warning: This email has been sent from outside KFUPM. Do not open links or attachments unless you are sure they are safe.

ZjQcmQRYFpfptBannerEnd

Hi Maher,

Having a test server will be useful but the fact that your test server (running the same version of RHEL and EPrints) has no problem but your production server does suggest an underlying issue rather than an issue with EPrints itself.  This means it is unlikely I will be able to help and unless anyone else has experienced the same error message running similar versions as you, then it is unlikely anyone will have an answer.  At best they may be able to recommend how to use memory management monitoring tools (e.g Valgrind), which typically require low level architecture knowledge to make sense of.

Unfortunately, I think finding a fix is going to require things like comparing your whole Apache configuration to check things like if there are any different modules loaded or different version of a module.  You can find (a sorted list) what modules are loaded by running:

httpd -M | sort

Unfortunately, this will not tell you the versions but as mod_perl is a likely candidate to be at fault if it is running a buggy version, it is worth running the following to see what version you have installed on both servers:

rpm -qa | grep mod_perl

Regards

David Newman

On 06/01/2025 10:56, Maher Abdellatif Ahmad Qahwash wrote:

CAUTION: This e-mail originated outside the University of Southampton.

Hi David

 

We have checked the access_log and ssl_access_log and nothing looks unusual.  In addition upgrading to 9.5 on our Test server running the same exact ePrints was successful and httpd started without the "double free or corruption" error ?? we don’t have any custom Apache modules used and we also compared the modules used on the Test server (which is working fine on 9.5) and the Production and the modules being loaded are exactly the same.

 

Any other ideas?

 

Many thanks

Maher

 

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Monday, January 6, 2025 12:20 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

 

Hi Maher, My first suggestion would have been that you were running mpm_event_module but as you are running mpm_prefork_module a core dump is an unusual problem. However, the "double free or corruption" error message is not what I

ZjQcmQRYFpfptBannerStart

Warning: This email has been sent from outside KFUPM. Do not open links or attachments unless you are sure they are safe.

ZjQcmQRYFpfptBannerEnd

Hi Maher,

My first suggestion would have been that you were running mpm_event_module but as you are running mpm_prefork_module a core dump is an unusual problem.  However, the "double free or corruption" error message is not what I remember seeing as a result of using mpm_event_module.

It sounds like this problem does not happen as soon as you start Apache but some time later.  Can you check your access_log and ssl_access_log to see if there are any unusual requests around this time?  Also, it sounds like this was not an issue when running RHEL 9.4 but is on 9.5.  I am not aware of anything that would have changed (most likely with Apache or mod_perl) that would have led to this issue but I do not manage any old (April 2019) versions of EPrints on EL 9.5.  As well as checking the access logs it might be worth checking all error log files for Apache.  I have found that critical errors often appear in ssl_error_log but warnings will appear in error_log and this can sometimes explain what is causing the critical issue.

If this does not provide any useful paths of investigation I would consider upgrading to the latest version of EPrints (3.4.6), as I know this works on EL 9.5.  I don't know if it would be possible to further diagnose this issue without running tools like strace to see what is happening in the lead up to the core dump.  Are you using any uncommon modules for Apache, as reading around the error message, it suggests that a badly coded Apache module may be the reason for the "double free or corruption" error message.

I also believe EPrints should run without seg-faulting and core dumping if using mpm_event_module if you upgrade to 3.4.6, as we fixed the issue that was causing this.  It does still suffer from a small amount of memory leak, so I would advise adding the following to your Apache config:

MaxConnectionsPerChild 1000

This ensures that workers are regularly restarted, freeing up an memory lost to leaks.  I have been running a busy EPrints 3.4.6 instance with MPM event on EL 9 (currently 9.5) with the above Apache configuration for several months now without any real issues.

Regards

David Newman

On 06/01/2025 8:47 am, Maher Abdellatif Ahmad Qahwash wrote:

CAUTION: This e-mail originated outside the University of Southampton.

Hi

 

We are  running EPrints 3.4.1-rc2 and have upgraded the Redhat from 9.4 to 9.5 but the httpd service is giving the following error which causes the service to stop:

 

[Mon Jan 06 11:11:00.536240 2025] [core:notice] [pid 125497:tid 125497] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

double free or corruption (out)

[Mon Jan 06 11:11:02.542042 2025] [core:notice] [pid 125497:tid 125497] AH00051: child pid 125498 exit signal Abort (6), possible coredump in /etc/httpd

double free or corruption (out)

[Mon Jan 06 11:11:05.550601 2025] [core:notice] [pid 125497:tid 125497] AH00051: child pid 125499 exit signal Abort (6), possible coredump in /etc/httpd

double free or corruption (out)

 

Please note that we are using the mpm_prefork_module and have the mpm_event_module commented out as it causes fragmentation and consumes very high CPU.

 

Any ideas what is causing the error?

 

Thanks

Maher

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.

 

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.