Hi Lizz

With data of this scale, I'd recommend getting it onto the eprints server first (scp and md5sum are your friends), and then attaching from the command line using an API script.  That way you have full control, and can be sure that it won't fail for network reasons.  See:

...for code that attaches a file to an eprint.  This script is my solution to a not-too-dissimilar problem, where I was batch creating eprints, and attaching images to the records (there were lots of them and the tiffs were really big).  My first step was to get everything on the server in some directory somewhere.

I don't think EPrints has a graceful way of attaching large files.  

--

Adam

Hi all,

 

I’m looking at options for importing files with records, and have read quite a few threads and bits of documentation, so have managed to get this to work in a couple of ways:

 

1)      Base64 encoding the files and including them in the import file

2)      Using the command line interface to import the files by referring to their location

3)      Temporarily allowing file imports and importing them by reference in the web interface

 

All of these work until I try and do the same from my normal PC to an instance of EPrints running on a server, at which point the import doesn’t have access to my local files so isn’t able to upload them.  I can’t change the access permissions on the files.  Base64 encoding is very much my option of last resort as some of the files this is for are expected to be around 10GB each, so storing a second (larger) copy of them in that format is going to put quite a strain on the storage available to the researcher.

 

Before I spend ages trying to get a workable solution, has anyone had a similar issue and found a workaround?

 

Lizz

 

--

Lizz Jennings BA MSc ACLIP MCLIP (Revalidated 2015)

Technical Data Officer

The Library 4.10, University of Bath, Bath, BA2 7AY UK

Ext. 3570 (External 01225 383570)

E.Jennings@bath.ac.uk

Research Data Management: http://www.bath.ac.uk/research/data

 

 

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/