EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #09816

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Ubuntu - Indexer as a service

CAUTION: This e-mail originated outside the University of Southampton.
Hi James,

I was having a similar problem. Our servers are periodically rebooted overnight. The problem we had was that the Indexer was killed instead of being stopped gracefully, so the PID file remained in place after reboot, preventing the Indexer from starting up again.

In the end I resorted to removing epindexer from /etc/init.d and installing a systemd service instead.

Problem was, we use the Indexer for more than just indexing. Some of the tasks are complex and prone to failure, so we tend to use the "--clear" or "--retry" options to the indexer script quite a lot. But manually stopping and starting the Indexer when systemd expects to control it is problematic.

(It would be nice if we could clear/retry tasks in bulk without restarting the Indexer; is there a reason why that would be a bad idea?)

In the end I wrote a helper script that I now use instead of calling the indexer script directly. It looks like this:

#!/usr/bin/env bash
case $1 in
	start)
		sudo systemctl start epindexer
		;;
	stop)
		sudo systemctl stop epindexer
		;;
	restart)
		sudo systemctl restart epindexer
		;;
	clear)
		sudo systemctl set-environment EPINDEXER='--clear'
		sudo systemctl restart epindexer
		sudo systemctl unset-environment EPINDEXER
		;;
	retry)
		sudo systemctl set-environment EPINDEXER='--retry'
		sudo systemctl restart epindexer
		sudo systemctl unset-environment EPINDEXER
		;;
	*)
		cat <<-END
		(help message)
		END
		;;
esac

And in the epindexer.service file:

[Service]
ExecStart=/opt/eprints3/bin/indexer start $EPINDEXER
ExecStop=/opt/eprints3/bin/indexer stop

Since using this solution I haven't had the issue of the PID file persisting on reboot. Systemd is supposed to take care of this with when PIDFile is set: "The service manager will not write to the file configured here, although it will remove the file after the service has shut down if it still exists."

A key thing to note is that the Indexer is a separate process to the Web interface, so you don't need to declare a dependency on Apache in the service unit, but you do need to declare the dependency on the database.

Cheers,
Alex.

On Wed, 2024-08-07 at 16:00 +0100, James Kerwin wrote:
Wondering if anyone can check if this is a sane thing to do.

This has been bugging me for some time since we upgraded from Ubuntu 14 to 20/22 (I've lost track at the moment).

Whenever the server was restarted the eprints Indexer would stall. Then after an hour or so say "Indexer has stopped..."

I've been learning about systemd and turning /opt/eprints3/epindexer into a service.

I wanted it to be bound to Apache, which hasn't quite worked. I got it to a point where issuing: systemctl [stop/start] epindexer on the command line would give the desired action. When Apache stopped the indexer would stop (e.g. the indexer.pid file in /opt/eprints3/var would disappear). Unfortunately it didn't restart when Apache was started (although it did at least tell me it wasn't running, which is a help in itself).
-- 
Alex Ball (he/him)
Research Data Librarian (Systems)
University of Bath, Bath BA2 7AY, UK
http://www.bath.ac.uk/library/