EPrints Technical Mailing List Archive

Message: #05161

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: IRStats2: incremental cron job

Thanks for the warnings about O - much appreciated.  It will require a little trial and error, I think.

Cheers

George

> -----Original Message-----
> From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-
> bounces@ecs.soton.ac.uk] On Behalf Of Field A.N.
> Sent: 23 November 2015 11:41
> To: eprints-tech@ecs.soton.ac.uk
> Subject: [EP-tech] Re: IRStats2: incremental cron job
> 
> Experiment and see how it goes, but bear in mind the running the time is
> something like:
> 
> 	((O+(PxA))*Dbl)/Dbp
> 
> 	O = Overhead (starting the process, etc) -- typically quite trivial, but
> can be significant if Dbl is high
> 	P = Time to process a single access line
> 	A = Number of accesses
> 	Dbl = The load on the database -- loosely, how many different things
> the database is currently doing. My gut feeling is that this is the one that is
> killer.
> 	Dbp = Database performance -- Your database server's hardware is
> the critical factor with this one
> 
> By running more frequently, you're going to increase the total amount of
> time the stats process is running because of O.  By running it at all the times
> of the day, you'll be inevitably running during times when Dbl is high
> (because people are doing things with the database).  I think your best bet is
> working out when Dbl is lowest (likely when the sun is over the Pacific
> Ocean), and running it then -- you will need to coordinate this with other
> tasks, perhaps setting browse view timeouts higher than 24 hours and
> generating them on a view-by-view basis on a staggered schedule).  The
> other way to sort things out is to increase Dbp, but that would probably need
> infrastructure upgrades.
> 
> --
> Adam Field
> Business Relationship Manager and Community Lead EPrints Services
> +44 (0)23 8059 8814
> 
> 
> 
> 
> 
> On 20 Nov 2015, at 16:32, George Macgregor wrote:
> 
> > Hi Adam - Thanks for the reply. Nice one - yes,  I did wonder if the job
> simply picked up where it left off last time...  I'll give it a bash!
> >
> > Cheers
> >
> > George
> >
> > -----Original Message-----
> > From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-
> bounces@ecs.soton.ac.uk] On Behalf Of Field A.N.
> > Sent: 20 November 2015 16:25
> > To: eprints-tech@ecs.soton.ac.uk
> > Subject: [EP-tech] Re: IRStats2: incremental cron job
> >
> > (b) should be possible -- the process just processes everything that hasn't
> yet been processed.
> >
> >
> > --
> > Adam Field
> > Business Relationship Manager and Community Lead EPrints Services
> > +44 (0)23 8059 8814
> >
> >
> >
> >
> >
> > On 19 Nov 2015, at 16:36, George Macgregor wrote:
> >
> >> Hello all,
> >>
> >> Just seeking some opinions from the list on an IRStats2 related issue. Like
> many of you we have IRStats2 installed (on 3.3.13).  We can observe from
> systems monitoring tools that the incremental processing cron job
> (configured to run over night in the small hours) can be extremely resource
> hungry and can often grind ePrints to a halt, e.g. excessively long response
> times for users while the job is running.  It is difficult to know how big a
> problem this is because the job runs when fewest people are known to
> access it.  Having said that, every download can be significant and I was
> wondering whether a) this also happens to be an issue for others, and b)
> whether anyone has attempted staggering the cron job, e.g. running it
> several times a day so that the job is less intrusive?  Not sure (b) is even
> possible without re-engineering the plugin but thought I would ask anyway!
> >>
> >> We are also reviewing our technical set-up because I think we are little
> tight on memory (4GB).
> >>
> >> Thanks in advance for any replies!
> >>
> >> Cheers
> >>
> >> George
> >> ----------------------------------
> >> George Macgregor | Information Services Directorate Curran Building |
> >> University of Strathclyde
> >> Tel: 0141 548 3496
> >> Email: george.macgregor@strath.ac.uk
> >> Web: http://personal.strath.ac.uk/george.macgregor/
> >> ----------------------------------
> >> The University of Strathclyde is a charitable body, registered in
> >> Scotland, with registration number SC015263
> >>
> >>
> >>
> >> *** Options:
> >> http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> >> *** Archive: http://www.eprints.org/tech.php/
> >> *** EPrints community wiki: http://wiki.eprints.org/
> >> *** EPrints developers Forum: http://forum.eprints.org/
> >
> >
> > *** Options:
> > http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> > *** Archive: http://www.eprints.org/tech.php/
> > *** EPrints community wiki: http://wiki.eprints.org/
> > *** EPrints developers Forum: http://forum.eprints.org/
> >
> > *** Options:
> > http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> > *** Archive: http://www.eprints.org/tech.php/
> > *** EPrints community wiki: http://wiki.eprints.org/
> > *** EPrints developers Forum: http://forum.eprints.org/
> 
> 
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: http://www.eprints.org/tech.php/
> *** EPrints community wiki: http://wiki.eprints.org/
> *** EPrints developers Forum: http://forum.eprints.org/