EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #07873

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] generate_views

Hi all,

This issue is also due to the need to generate citations for each item
in a browse view which can be slow, as each one can require a number of
SQL queries.  The citation caching code I describe in the
Hyperauthoship thread would like improve this significantly and was one
of the main reason I was writing it.

I have not had a chance to look at this again since I received Chris'
feedback.  When I do I will hopefully be able to provided an updated
version for people to try.  Unfortunately, this is still a long way
upstream and requires changes to the core codebase, so is not something
that will solve your problem in the short term.

Regards

David Newman



On Wed, 2019-06-05 at 19:00 +1000, Matthew Kerwin via Eprints-tech
wrote:
> We've put a lot of energy into this problem over the years. I've
> variously: overridden the default Search generation for some
> problematic views with custom SQL queries, and added a wrapper around
> the generate_views script that runs four instances in parallel (one
> for each core on our server).  It still takes a couple of hours every
> week, but it's no longer 12+ hours.
>
> I'm not at work right now to get my hands on the code.  I'll see if I
> can get to it tomorrow.
>
> Cheers
>
> On Wed, 5 Jun 2019 at 18:10, James Kerwin via Eprints-tech
> <eprints-tech@ecs.soton.ac.uk> wrote:
> >
> >
> > Hi All,
> >
> > The generate_views script runs on our EPrints server once a week
> > (early hours of Saturday morning).
> >
> > It started on Saturday and is still running now. I'm confident that
> > it's still generating views, there are just a lot of records to go
> > through. Has anybody experienced anything similar or have
> > suggestions on how I could decrease how long it takes?
> >
> > I think my main option is to get rid of some views (not too keen on
> > that) or speak to our computing services department and ask for the
> > server to be given a bit of a boost (likely to be rejected unless
> > our EPrints instance is running so slowly it goes backwards).
> >
> > I suppose I could run the script every two weeks etc. but I already
> > feel as though a week is quite a long time between the views being
> > updated. Plus a minimum of 4 days is a long time for this script to
> > be running.
> >
> > Does anybody have experience with this and any advice on what I
> > could do?
> >
> > Thanks,
> > James
> > *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprint
> > s-tech
> > *** Archive: https://eur03.safelinks.protection.outlook.com/?url=ht
> > tp%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7Cdrn%40e
> > cs.soton.ac.uk%7C2edaa07a617c471ef78f08d6e9947519%7C4a5378f929f44d3
> > ebe89669d03ada9d8%7C0&amp;sdata=Uh01EaUA%2FvQirGHdMzLtPV5W5eye4b31l
> > PH4875sCqk%3D&amp;reserved=0
> > *** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Feur03.safelinks.protection.outl&amp;data=01%7C01%7C%7C67f29c025b9e4b0f9ceb08d6e99683a8%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=dG4ijNMHwcS%2BVYUko%2B08bXHvUJ5bk%2BdzEPtB44lSRQg%3D&amp;reserved=0
> > ook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7Cdr
> > n%40ecs.soton.ac.uk%7C2edaa07a617c471ef78f08d6e9947519%7C4a5378f929
> > f44d3ebe89669d03ada9d8%7C0&amp;sdata=h93CiORhMNLHfZNGFoOik%2F0M%2B8
> > Y%2FWdF%2BrQz7TUqluus%3D&amp;reserved=0
> > *** EPrints developers Forum: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Feur03.safelinks.protection.ou&amp;data=01%7C01%7C%7C67f29c025b9e4b0f9ceb08d6e99683a8%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=1kOgzdsDz8QhU5oPc8hKQz5he5%2Bo78b61uikrBeTkp8%3D&amp;reserved=0
> > tlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&amp;data=01%7C01%7
> > Cdrn%40ecs.soton.ac.uk%7C2edaa07a617c471ef78f08d6e9947519%7C4a5378f
> > 929f44d3ebe89669d03ada9d8%7C0&amp;sdata=WScn%2FXKfHsDnCu%2BD3soh%2F
> > 5vzmSk2dAio8c5FARMIUU4%3D&amp;reserved=0
>
>