EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #03680

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: load throttling strategy

Il 15/12/2014 16:35, Ian Stuart ha scritto:

I bet he's suffering the problem I had: Google loads a 'view' page,
which has hit it's "out of date" time so rebuilds.
In the mean time, another httpd catches the next google page load, which
has also hit its "out of date" time, so spawns ANOTHER page rebuild.


doesn't eprints see it is already rebuilding?



It very quickly ramps up, and WILL swamp the server... and WILL crash
the server.

On 15/12/14 15:21, sf2 wrote:

Which issue are you trying to solve by introducing such a delay?

Seb

On 15.12.2014 15:17, Gooding R.W. wrote:


Hi,

does anyone have an easy way to throttle eprints website crawlers ?

I’m wondering is something as a short delay for each ePrint accessed -
the delay being proportional to the current load ?  That’s perhaps a
little blunt but might it be very easy to implement ?