EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #05160

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: IRStats2: incremental cron job

Experiment and see how it goes, but bear in mind the running the time is something like:

	((O+(PxA))*Dbl)/Dbp

	O = Overhead (starting the process, etc) -- typically quite trivial, but can be significant if Dbl is high
	P = Time to process a single access line
	A = Number of accesses
	Dbl = The load on the database -- loosely, how many different things the database is currently doing. My gut feeling is that this is the one that is killer.
	Dbp = Database performance -- Your database server's hardware is the critical factor with this one

By running more frequently, you're going to increase the total amount of time the stats process is running because of O.  By running it at all the times of the day, you'll be inevitably running during times when Dbl is high (because people are doing things with the database).  I think your best bet is working out when Dbl is lowest (likely when the sun is over the Pacific Ocean), and running it then -- you will need to coordinate this with other tasks, perhaps setting browse view timeouts higher than 24 hours and generating them on a view-by-view basis on a staggered schedule).  The other way to sort things out is to increase Dbp, but that would probably need infrastructure upgrades.

--
Adam Field
Business Relationship Manager and Community Lead
EPrints Services
+44 (0)23 8059 8814





On 20 Nov 2015, at 16:32, George Macgregor wrote:

> Hi Adam - Thanks for the reply. Nice one - yes,  I did wonder if the job simply picked up where it left off last time...  I'll give it a bash!
> 
> Cheers
> 
> George
> 
> -----Original Message-----
> From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of Field A.N.
> Sent: 20 November 2015 16:25
> To: eprints-tech@ecs.soton.ac.uk
> Subject: [EP-tech] Re: IRStats2: incremental cron job
> 
> (b) should be possible -- the process just processes everything that hasn't yet been processed.
> 
> 
> --
> Adam Field
> Business Relationship Manager and Community Lead EPrints Services
> +44 (0)23 8059 8814
> 
> 
> 
> 
> 
> On 19 Nov 2015, at 16:36, George Macgregor wrote:
> 
>> Hello all,
>> 
>> Just seeking some opinions from the list on an IRStats2 related issue. Like many of you we have IRStats2 installed (on 3.3.13).  We can observe from systems monitoring tools that the incremental processing cron job (configured to run over night in the small hours) can be extremely resource hungry and can often grind ePrints to a halt, e.g. excessively long response times for users while the job is running.  It is difficult to know how big a problem this is because the job runs when fewest people are known to access it.  Having said that, every download can be significant and I was wondering whether a) this also happens to be an issue for others, and b) whether anyone has attempted staggering the cron job, e.g. running it several times a day so that the job is less intrusive?  Not sure (b) is even possible without re-engineering the plugin but thought I would ask anyway!
>> 
>> We are also reviewing our technical set-up because I think we are little tight on memory (4GB).
>> 
>> Thanks in advance for any replies!
>> 
>> Cheers
>> 
>> George
>> ----------------------------------
>> George Macgregor | Information Services Directorate Curran Building | 
>> University of Strathclyde
>> Tel: 0141 548 3496
>> Email: george.macgregor@strath.ac.uk
>> Web: http://personal.strath.ac.uk/george.macgregor/
>> ----------------------------------
>> The University of Strathclyde is a charitable body, registered in 
>> Scotland, with registration number SC015263
>> 
>> 
>> 
>> *** Options: 
>> http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
>> *** EPrints developers Forum: http://forum.eprints.org/
> 
> 
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: http://www.eprints.org/tech.php/
> *** EPrints community wiki: http://wiki.eprints.org/
> *** EPrints developers Forum: http://forum.eprints.org/
> 
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: http://www.eprints.org/tech.php/
> *** EPrints community wiki: http://wiki.eprints.org/
> *** EPrints developers Forum: http://forum.eprints.org/