EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #10418

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Links in PDFs that don't contain a protocol, resulting in lots of 'hits'

CAUTION: This e-mail originated outside the University of Southampton.
Hi All,
I've been looking at some 'intersting' traffic patterns in our repository and traced some of them back to PDFs that have been crawled, but also have poorly formatted links in them.
From an intitial PDF downloaded from 
    /id/eprint/[eprintid]/[pos]/Document.pdf
the 'bad' links results in the crawler trying to access links within the downloaded PDF relatively to the PDF URL. In the logs these appear like:
    GET /id/eprint/[eprintid]/[pos]/www.example.com/some_page/

EPrints responds to the above request with the PDF again (based on eprintid and pos). The crawler then uses that as a base, so the next 'round' from the crawler requests:
    GET /id/eprint/[eprintid]/1/www.example.com/www.example2.com/some_other_page/

This real example from my logs shows 8-iterations of this behaviour (I've changed the initial domain, so their site doesn't get crawled with this URL - that would be unfair):
www.housinglin.org.uk.example.com/_assets/Resources/Housing/OtherOrganisation/www.ageingwellinwales.com/Libraries/Documents/www.alz.co.uk/research/www.ageingwellinwales.com/Libraries/Documents/www.ssehsactive.org.uk/userfiles/Documents/www.gov.uk/government/uploads/system/uploads/attachment_data/file/215101/www.ageingwellinwales.com/Libraries/Documents/www.mrc.ac.uk/documents/pdf/mrc-phsrn-process-evaluation-guidance-final

Have others noted similar behaviour? Is there a way to limit this behaviour?
The PDFs aren't generally in our control - they're author accepted manuscripts, so correcting the PDF somehow isn't a realistic option.

Cheers,
John

John Salter

https://orcid.org/0000-0002-8611-8266

 

White Rose Libraries Technical Officer
Library and Research Management team, IT
University of Leeds