All,

 

I’ve noticed that we are getting crawled by what seems to be a newer robot “AhrefsBot” (http://ahrefs.com) That also seems to be ignoring the “Disallow: /cgi/” stanza as when looking at the logs or the apache server-status it is hitting things in /cgi.  L

 

As a first measure to reign this bot in I’d like to add a parameter to the default robots.txt file “Crawl-Delay: 2” per their documentation (https://ahrefs.com/robot/) but not finding a simple way of doing in EPrints so I started to go through the files and ran across: perl_lib/EPrints/Apache/RobotsTxt.pm where I see what is the default definition for the robots.txt file.  I’ve updated that file and restarted the web server but alas the robots.txt file does not change. 

 

So two questions:

1.       Does anyone have a hint on what needs to be done to identify a new bot correctly?  I’ve also found the perl_lib/robots.pm but not sure where to add the AhrefsBot to the file.

2.       Does anyone know how to update the robots.txt file?  Is it per archive?

 

Thanks,

Brian Gregg.

 

 

Brian D. Gregg

Solutions Architect | Manager Systems Development

University of Pittsburgh | University Library System

Address: 7500 Thomas Blvd.  Room 129 Pittsburgh, PA 15208

Tel: (412) 648-3264 | Email: bdgregg@pitt.edu | Fax: (412) 648-3585