We’re using SWORD with epdata packages to deposit documents and multimedia into our repository (3.2). This works fine for small file sizes but at CPU and memory increases quickly until with a ~200MB file the httpd process consumes all available memory and dies.  This is on a RHEL5 64bit box with 8GB memory with a separate DB server.

Clearly, the epdata format is not the most appropriate for this size file due to the increased file size as a result of the base64 encoding and because the document is embedded within the XML.  Changing package format may alleviate/resolve the problem but as this is definitely going to be a challenge in our environment I’m hoping it will be easier to deal with the issue within EPrints.

Note, I’ve already ascertained that is not related to libxm2’s XML_PARSE_HUGE option being disabled, the failure occurs trying to run df.

I’m about to start hunting for memory leaks and then doing additional memory profiling.  If anyone has any suggestions about likely locations for memory leaks in the code, information about expected memory usage for SWORD with epdata packages, data from previous profiling, etc, it would be very valuable.

Cheers

Mark

Mark Gregson | Applications and Development Team Leader
Library eServices | Queensland University of Technology
Level 3 | R Block | Kelvin Grove Campus | GPO Box 2434 | Brisbane 4001
Phone: +61 7 3138 3782 | Web: http://eprints.qut.edu.au/
ABN: 83 791 724 622
CRICOS No: 00213J