Hi Admin - I wish to Unsubscribe. Thank you.

Hi Tomasz,

I can only theorise at why /id/dump exists.  My best theory is that a "semantic" third party application wants to download the metadata for all eprints in the live archive or possibly even all the documents of those eprints.  I did note that some older repositories will serve RDF XML rather than a list of IDs, which is why I used "semantic".  I think more recently repositories (3.4.7+) that have cfg/flavours/pub_lib/cfg.d/y_export_privacy.pl will serve IDs because RDF XML is not publicly accessible by default as it is considered as a format that may contain private data.

If you are on an older repository that serves RDF XML, I found that around 30,000 live archive records took around 2 minutes 30 seconds to generate and download.  This would likely be even longer in a web browser, which may try to render the XML.  When you get served IDs this is a lot quicker, only taking around 3-4 seconds for repository of around 30,000 records (set the accept header to text/plain if you want to try this).

A plan may be to hide /id/dump and anything similar behind requests that require an Authorization header or a session cookie, (i.e. a logged in user).  

/id/repository does not work on 3.4.7+ repositories and gives a 500 error.  This is due to cfg/flavours/pub_lib/cfg.d/y_export_privacy.pl not allowing access to the only format that has an output_graph function that can be called, which is presumably necessary to generatye

/id/subject only works for me if you specify a subject ID like id/subject/GN.  On 3.4.7+ repositories you need to have authenticated/authorized to get a useful response.  In both cases it generally seems to provide the who purposed subject tree (e.g. Library of Congress or your institutions divisional structure).

There are probably one simple-ish change that should be made to the next version of EPrints, hide /id/dump and /id/repository behind authentication.  Because the user would need to be authenticated, that should mean that these would then no longer be a privacy concern and could return RDF XML.  Subjects are already authentication restricted, so no change is required for these.  On top of this is may be worth providing configuration, which by default disabled sitemap-sc.xml.  Then in the off chance it is still needed it could be re-enabled.

That all being said, I really do feel that the best option may be to rip it all out, as I would be amazed if any of the requests are essential to any EPrints repository today.  However, you only ever find this out some way down the line, when someone asks why does this not work anymore.  This is why it is more acceptable in a major version change (e.g. 3.4 to 3.5) than a minor one, (e.g. 3.4.7 to 3.4.8).

Regards

David Newman

On 11/08/2026 15:51, Tomasz Neugebauer wrote:

CAUTION: This e-mail originated outside the University of Southampton.

Hi David,

 

Thanks for the detailed answer!

The reason I was concerned about the sitemap is that it is out there in the open, so I've become cautious about giving bots access to a request URL that is dynamic and computationally resource heavy, requiring a lot of CPU / RAM / database queries to complete each and every time it is requested.  The "id/dump" part of the sitemap seems to fall in that category.  What is it for?  When I ask for it, the server takes a long time to dump out a whole bunch of IDs. If we can't rationalize why it's there, then it shouldn't be there, for everyone, by default, with the only way to override being changing EPrints core perl apache module code. Having said that, in the meantime, I appreciate knowing that is what I need to do.  What about those other parts of the sitemap (id/repository and id/subject), are they also dynamic and computationally resource heavy? Are they needed if a sitemap is being generated with generate_sitemap script?

 

Tomasz

 

 

 

________________________________________________

Tomasz Neugebauer
Senior Librarian | Bibliothécaire titulaire
Digital Projects & Systems Development Librarian / Bibliothécaire des Projets Numériques & Développement de Systèmes
Concordia University / Université Concordia

Tel. / Tél. 514-848-2424 ext. / poste 7738
Email / courriel: tomasz.neugebauer@concordia.ca

Mailing address / adresse postale: 1455 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8
Street address / adresse municipale: 1400 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8

library.concordia.ca

 

---

From: David R Newman <drn@ecs.soton.ac.uk>
Sent: August 11, 2026 2:59 AM
To: eprints-tech@ecs.soton.ac.uk <eprints-tech@ecs.soton.ac.uk>; Tomasz Neugebauer <Tomasz.Neugebauer@concordia.ca>
Subject: Re: [EP-tech] sitemap generation in EPrints

 

Attention This email originates from outside the concordia.ca domain. // Ce courriel provient de l'extérieur du domaine de concordia.ca

 

 

Hi Tomasz,

sitemap-sc.xml is very much a legacy thing, which was implemented well before my time.  I have been loathed to remove it, in case it served some important purpose for which I was unaware.  Feel free to add Apache configuration that sits above EPrints to either 404 this or maybe just redirect to sitemap.xml.  If you want to add this as an issue on GitHub [1], then we can consider what should be done about sitemap-sc.xml and maybe also access to /id/dump.

I wrote the generate_sitemap script.  Originally, this was just for EPrints repositories I helped maintain.  I then rolled this into the main codebase [1] at a later point, when I had seen it was useful.  In particular, using it on Google's search admin console to make sure publications in EPrints were as indexed as comprehensively as possible.  

I agree that use generate_sitemap could be better documented, (i.e. you need to setup a cron job for the sitemap to be generated / regularly regenerated). As it grew quite organically, there was not a tipping point where I thought this is now standard functionality and therefore needs to be documented as such.  I will look into updating the wiki in suitable places.

You made an allusion to customisation.  I was not sure if that was just a reference to sitemap-sc.xml and removing /id/dump or if you wanted to be able to also customise sitemap.xml, as you wanted links other than to abstract pages in your sitemap.  If it is the former, then I think unfortunately, you would need to edit perl_lib/EPrints/Apache/SiteMap.pm either in place or a copy in site_lib/plugins/ to remove /id/dump.

If it is the latter, as sitemap.xml links to secondary files to avoid having more than 1000 entries per file, (as preferred by Google's search admin console), then it would not be too difficult to customise to get the script to detect a file called something like extra.xml.  You could then maintain your additional sitemap links in this file (under the archive's cfg/static/sitemaps/ directory) and this would be made available with the numbered sitemap files.  I would be happy to create a GitHub issue to look into such customisation.

Regards

David Newman

 

 

 

[1] https://github.com/eprints/eprints3.4

On 10/08/2026 22:04, Tomasz Neugebauer wrote:

CAUTION: This e-mail originated outside the University of Southampton.

CAUTION: This e-mail originated outside the University of Southampton.

As I keep working on refining our apache modsecurity WAF to protect our repository from bots, and to adding anubis, there are also two more obvious points we need to talk about:

 

1. EPrints sitemap generation

2. EPrints robots.txt

 

I welcome any comments on the robots.txt, but for this message, can we please talk about the sitemap?  I'm confused about it.  There is a "default" sitemap that is generated that is referred to in the code as something like "_insert_semantic_web_extensions" (dating back to eprints 3.2) and it generates something by default that is a "sitemap-sc.xml" file or just a regular sitemap.xml file (if you didn't run the generate sitemap bin script). In either case, the "-sc.xml" part of the sitemap is hard coded into the Apache handling, forcing always a sitemap that includes a query for a full generic "id/dump" which seems super intense, computationally. 

 

We need better documentation, instruction on how to generate sitemaps in EPrints.  What if I don't want the default links that are generated into the "sitemap-sc.xml" or "sitemap.xml"?  In particular, I'm a little concerned about bots hitting that generic "id/dump"... where would I customize this (and is there some important reason not to)?

 

Best wishes,

 

Tomasz

 

 

________________________________________________

Tomasz Neugebauer
Senior Librarian | Bibliothécaire titulaire
Digital Projects & Systems Development Librarian / Bibliothécaire des Projets Numériques & Développement de Systèmes
Concordia University / Université Concordia

Tel. / Tél. 514-848-2424 ext. / poste 7738
Email / courriel: tomasz.neugebauer@concordia.ca

Mailing address / adresse postale: 1455 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8
Street address / adresse municipale: 1400 De Maisonneuve Blvd. W., LB-540-03, Montreal, Quebec H3G 1M8

library.concordia.ca

 

*** Options: https://wiki.eprints.org/w/Eprints-tech_Mailing_List

*** Archive: https://www.eprints.org/tech.php/

*** EPrints community wiki: https://wiki.eprints.org/