EPrints Technical Mailing List Archive

Message: #08126

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Server timeouts - MySQL Problems

Hi James,

Technically, there is also mtop (m for mysql) but I don't think it is in the standard package repositories and is only useful for keeping an eye on something at the time it is happening, unless you can hack a way to get it to log like atop.

Regards

David Newman

On 05/03/2020 14:37, James Kerwin wrote:
Thanks David!

I've used top and htop, but never atop. I've just installed it so I will get to investigating it now. Sounds like it could be really useful. Improves on my previous idea of staying up until the suspected failure time and looking at which processes were running.

I'll get working on point two now.

I may delay point three until I'm really stuck. Although I have just noticed that the Elements "get_records" script appears to run for longer than an hour, so it's for example still running the 1pm script when the 2pm script starts.

I'm trying to decide if there's any great harm in doing frequent curl calls to the homepage from another server to see at which point it fails so I can pin down a more precise time for the problem.

So much to investigate!

Thanks again for your advice. It's greatly appreciated.

James



On Thu, Mar 5, 2020 at 1:37 PM Newman D.R. <drn@ecs.soton.ac.uk> wrote:

Hi James,

Several suggestions:

1. Try install atop [1], this creates log files similar to what you get from running the top command.  This will allow you to look back later to see what was going on at the times when the server was not responding.  By default it takes a snapshot every 10 minutes.  It might be worth swaping this to every minute or couple of minutes.

2. Edit MySQL's configuration to introduce a log file for slow running queries [2].

3. I use something called pt-kill to kill very long running queries that may be blocking other queries [3].

Regards

David Newman

[1] https://linux.die.net/man/1/atop

[2] https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html

[3] https://www.percona.com/doc/percona-toolkit/LATEST/pt-kill.html


On 05/03/2020 12:46, James Kerwin via Eprints-tech wrote:
Hi All,

This isn't necessarily directly EPrints related, but its about a server running EPrints.

I've noticed a problem this week with the repository. In the early hours of the morning the number of users drops to zero for several hours between 2am and 6am (according to Google Analytics). Due to having  a cold I've been up between these times and can confirm that the repository website times out when I try to connect from home.

I don't get any memory or CPU warnings from our monitoring software. My gut instinct is that it's an issue with MySQL connections not closing in a timely manner. We do have cron jobs that run at 1:30, 2:30 and 3:30 which I'm aware fall right within the problem zone, but these have been running at the same time for years and have never caused an issue.

Has anybody experienced anything similar to this or have suggestions as to how I could chase it down?

It's a Ubuntu server with MySQL running EPrints 3.3.14. I don't think it's an EPrints issue, but there is nothing in the log files to suggest what's happening. The apache error log is blank for the hours that the server won't connect.

Thanks,
James

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/

Virus-free. www.avg.com