EPrints Technical Mailing List Archive

Message: #05427

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: Importing files

Thanks Adam, although it’s a challenge given the storage available on my server (the repository file storage is external).  I think I may need to mount the drive (which now I’ve tried it seems to work well), but I shall take a look at the script for verification as that will mean I can consider some automation I think.

 

Lizz

 

--

Lizz Jennings BA MSc ACLIP MCLIP (Revalidated 2015)

Technical Data Officer

The Library 4.10, University of Bath, Bath, BA2 7AY UK

Ext. 3570 (External 01225 383570)

E.Jennings@bath.ac.uk

Research Data Management: http://www.bath.ac.uk/research/data

 

 

From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of Adam Field
Sent: 16 February 2016 23:02
To: eprints-tech@ecs.soton.ac.uk
Subject: [EP-tech] Re: Importing files

 

Hi Lizz

 

With data of this scale, I'd recommend getting it onto the eprints server first (scp and md5sum are your friends), and then attaching from the command line using an API script.  That way you have full control, and can be sure that it won't fail for network reasons.  See:

 

 

...for code that attaches a file to an eprint.  This script is my solution to a not-too-dissimilar problem, where I was batch creating eprints, and attaching images to the records (there were lots of them and the tiffs were really big).  My first step was to get everything on the server in some directory somewhere.

 

I don't think EPrints has a graceful way of attaching large files.  

 

--

Adam

 

 

On 16 Feb 2016, at 17:57, Lizz Jennings <e.jennings@bath.ac.uk> wrote:



Hi all,

 

I’m looking at options for importing files with records, and have read quite a few threads and bits of documentation, so have managed to get this to work in a couple of ways:

 

1)      Base64 encoding the files and including them in the import file

2)      Using the command line interface to import the files by referring to their location

3)      Temporarily allowing file imports and importing them by reference in the web interface

 

All of these work until I try and do the same from my normal PC to an instance of EPrints running on a server, at which point the import doesn’t have access to my local files so isn’t able to upload them.  I can’t change the access permissions on the files.  Base64 encoding is very much my option of last resort as some of the files this is for are expected to be around 10GB each, so storing a second (larger) copy of them in that format is going to put quite a strain on the storage available to the researcher.

 

Before I spend ages trying to get a workable solution, has anyone had a similar issue and found a workaround?

 

Lizz

 

--

Lizz Jennings BA MSc ACLIP MCLIP (Revalidated 2015)

Technical Data Officer

The Library 4.10, University of Bath, Bath, BA2 7AY UK

Ext. 3570 (External 01225 383570)

E.Jennings@bath.ac.uk

Research Data Management: http://www.bath.ac.uk/research/data

 

 

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/



Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.

Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.