As I keep working on refining our apache modsecurity WAF to protect our repository from bots, and to adding anubis, there are also two more obvious points we need to talk about:

I welcome any comments on the robots.txt, but for this message, can we please talk about the sitemap?  I'm confused about it.  There is a "default" sitemap that is generated that is referred to in the code as something like "_insert_semantic_web_extensions" (dating back to eprints 3.2) and it generates something by default that is a "sitemap-sc.xml" file or just a regular sitemap.xml file (if you didn't run the generate sitemap bin script). In either case, the "-sc.xml" part of the sitemap is hard coded into the Apache handling, forcing always a sitemap that includes a query for a full generic "id/dump" which seems super intense, computationally. 

We need better documentation, instruction on how to generate sitemaps in EPrints.  What if I don't want the default links that are generated into the "sitemap-sc.xml" or "sitemap.xml"?  In particular, I'm a little concerned about bots hitting that generic "id/dump"... where would I customize this (and is there some important reason not to)?

Best wishes,

Tomasz

________________________________________________

Tomasz Neugebauer
Senior Librarian | Bibliothécaire titulaire
Digital Projects & Systems Development Librarian / Bibliothécaire des Projets Numériques & Développement de Systèmes
Concordia University / Université Concordia

Tel. / Tél. 514-848-2424 ext. / poste 7738
Email / courriel: tomasz.neugebauer@concordia.ca

Mailing address / adresse postale: 1455 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8
Street address / adresse municipale: 1400 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8

library.concordia.ca