Attention This email originates from outside the concordia.ca domain. // Ce courriel provient de l'extérieur du domaine de concordia.ca

Hi Tomasz,

sitemap-sc.xml is very much a legacy thing, which was implemented well before my time.  I have been loathed to remove it, in case it served some important purpose for which I was unaware.  Feel free to add Apache configuration that sits above EPrints to either 404 this or maybe just redirect to sitemap.xml.  If you want to add this as an issue on GitHub [1], then we can consider what should be done about sitemap-sc.xml and maybe also access to /id/dump.

I wrote the generate_sitemap script.  Originally, this was just for EPrints repositories I helped maintain.  I then rolled this into the main codebase [1] at a later point, when I had seen it was useful.  In particular, using it on Google's search admin console to make sure publications in EPrints were as indexed as comprehensively as possible.  

I agree that use generate_sitemap could be better documented, (i.e. you need to setup a cron job for the sitemap to be generated / regularly regenerated). As it grew quite organically, there was not a tipping point where I thought this is now standard functionality and therefore needs to be documented as such.  I will look into updating the wiki in suitable places.

You made an allusion to customisation.  I was not sure if that was just a reference to sitemap-sc.xml and removing /id/dump or if you wanted to be able to also customise sitemap.xml, as you wanted links other than to abstract pages in your sitemap.  If it is the former, then I think unfortunately, you would need to edit perl_lib/EPrints/Apache/SiteMap.pm either in place or a copy in site_lib/plugins/ to remove /id/dump.

If it is the latter, as sitemap.xml links to secondary files to avoid having more than 1000 entries per file, (as preferred by Google's search admin console), then it would not be too difficult to customise to get the script to detect a file called something like extra.xml.  You could then maintain your additional sitemap links in this file (under the archive's cfg/static/sitemaps/ directory) and this would be made available with the numbered sitemap files.  I would be happy to create a GitHub issue to look into such customisation.

Regards

David Newman

[1] https://github.com/eprints/eprints3.4

CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

As I keep working on refining our apache modsecurity WAF to protect our repository from bots, and to adding anubis, there are also two more obvious points we need to talk about:

1. EPrints sitemap generation
2. EPrints robots.txt

I welcome any comments on the robots.txt, but for this message, can we please talk about the sitemap?  I'm confused about it.  There is a "default" sitemap that is generated that is referred to in the code as something like "_insert_semantic_web_extensions" (dating back to eprints 3.2) and it generates something by default that is a "sitemap-sc.xml" file or just a regular sitemap.xml file (if you didn't run the generate sitemap bin script). In either case, the "-sc.xml" part of the sitemap is hard coded into the Apache handling, forcing always a sitemap that includes a query for a full generic "id/dump" which seems super intense, computationally. 

We need better documentation, instruction on how to generate sitemaps in EPrints.  What if I don't want the default links that are generated into the "sitemap-sc.xml" or "sitemap.xml"?  In particular, I'm a little concerned about bots hitting that generic "id/dump"... where would I customize this (and is there some important reason not to)?

Best wishes,

Tomasz

________________________________________________

Tomasz Neugebauer
Senior Librarian | Bibliothécaire titulaire
Digital Projects & Systems Development Librarian / Bibliothécaire des Projets Numériques & Développement de Systèmes
Concordia University / Université Concordia

Tel. / Tél. 514-848-2424 ext. / poste 7738
Email / courriel: tomasz.neugebauer@concordia.ca

Mailing address / adresse postale: 1455 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8
Street address / adresse municipale: 1400 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8

library.concordia.ca

*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List
*** Archive: https://www.eprints.org/tech.php/
*** EPrints community wiki: https://wiki.eprints.org/