Hi David

We have checked the access_log and ssl_access_log and nothing looks unusual.  In addition upgrading to 9.5 on our Test server running the same exact ePrints was successful and httpd started without the "double free or corruption" error ?? we don’t have any custom Apache modules used and we also compared the modules used on the Test server (which is working fine on 9.5) and the Production and the modules being loaded are exactly the same.

Any other ideas?

Many thanks

Maher

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: Monday, January 6, 2025 12:20 PM
To: Maher Abdellatif Ahmad Qahwash <qahwash@kfupm.edu.sa>; eprints-tech@ecs.soton.ac.uk
Subject: Re: Redhat upgrade to 9.5

Hi Maher,

My first suggestion would have been that you were running mpm_event_module but as you are running mpm_prefork_module a core dump is an unusual problem.  However, the "double free or corruption" error message is not what I remember seeing as a result of using mpm_event_module.

It sounds like this problem does not happen as soon as you start Apache but some time later.  Can you check your access_log and ssl_access_log to see if there are any unusual requests around this time?  Also, it sounds like this was not an issue when running RHEL 9.4 but is on 9.5.  I am not aware of anything that would have changed (most likely with Apache or mod_perl) that would have led to this issue but I do not manage any old (April 2019) versions of EPrints on EL 9.5.  As well as checking the access logs it might be worth checking all error log files for Apache.  I have found that critical errors often appear in ssl_error_log but warnings will appear in error_log and this can sometimes explain what is causing the critical issue.

If this does not provide any useful paths of investigation I would consider upgrading to the latest version of EPrints (3.4.6), as I know this works on EL 9.5.  I don't know if it would be possible to further diagnose this issue without running tools like strace to see what is happening in the lead up to the core dump.  Are you using any uncommon modules for Apache, as reading around the error message, it suggests that a badly coded Apache module may be the reason for the "double free or corruption" error message.

I also believe EPrints should run without seg-faulting and core dumping if using mpm_event_module if you upgrade to 3.4.6, as we fixed the issue that was causing this.  It does still suffer from a small amount of memory leak, so I would advise adding the following to your Apache config:

MaxConnectionsPerChild 1000

This ensures that workers are regularly restarted, freeing up an memory lost to leaks.  I have been running a busy EPrints 3.4.6 instance with MPM event on EL 9 (currently 9.5) with the above Apache configuration for several months now without any real issues.

Regards

David Newman

CAUTION: This e-mail originated outside the University of Southampton.

Hi

 

We are  running EPrints 3.4.1-rc2 and have upgraded the Redhat from 9.4 to 9.5 but the httpd service is giving the following error which causes the service to stop:

 

[Mon Jan 06 11:11:00.536240 2025] [core:notice] [pid 125497:tid 125497] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

double free or corruption (out)

[Mon Jan 06 11:11:02.542042 2025] [core:notice] [pid 125497:tid 125497] AH00051: child pid 125498 exit signal Abort (6), possible coredump in /etc/httpd

double free or corruption (out)

[Mon Jan 06 11:11:05.550601 2025] [core:notice] [pid 125497:tid 125497] AH00051: child pid 125499 exit signal Abort (6), possible coredump in /etc/httpd

double free or corruption (out)

 

Please note that we are using the mpm_prefork_module and have the mpm_event_module commented out as it causes fragmentation and consumes very high CPU.

 

Any ideas what is causing the error?

 

Thanks

Maher

---

DISCLAIMER: The information in this email and its attachments (if any) is intended for the addressee only and may contain confidential or privileged information. If you are not the intended recipient, please delete the email and its attachments from your system and notify the sender immediately. You should not retain, disclose, copy, or use this email or any of its contents for any purpose, nor disclose its contents to any other person. KFUPM is not responsible for changes made to this message after it was sent. Statements and opinions expressed in this e-mail are those of the sender, and do not necessarily reflect those of KFUPM. KFUPM is not liable for any effect or virus damage caused by this message.
إن المعلومات الواردة في هذا البريد الإلكتروني ومرفقاته إن وجدت، قد تكون خاصة أو سرية؛ فإذا لم تكن المقصود بهذه الرسالة؛ فيُرجى منك حذفها ومرفقاتها من نظامك وإخطار المرسل بخطأ وصولها إليك فورا. كما لا يجوز نسخ أي جزء منها أو مرفقاتها ، أو الإفصاح عن محتوياتها لأي شخص أو استعمالها لأي غرض آخر. إن جامعة الملك فهد للبترول والمعادن لا تتحمل مسؤولية التغييرات التي يتم إجراؤها على هذه الرسالة بعد إرسالها. وإن البيانات أو الآراء المعبر عنها في هذا البريد، هي بيانات تخص مُرسلها، ولا تعكس بالضرورة رأي وبيانات الجامعة. كما لا تتحمل الجامعة مسؤولية أي تأثير ينتج عن هذه الرسالة أوعن أي فيروس قد تحمله.