EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #00789

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Re: Importing files from another system

That's fine with me :o)

I've been trying to get my head round the new codebase (I'm used to v<3.2. Now looking at 3.3), and trying to work out if the filename could get set automatically.

Running the import tool without a filename is broken.
It looks like all other bits of data work OK.
Below is the output when just a ...<file><url>file:///...</url></file>... is used.

I think this could be fixed by adding a test to Eprint::DataObj::File::create_from_data that checks that
$data->{filename} exists before using it. If it doesn't exist, extract it from $filepath (I'm guessing there may already be a function somewhere in EPrints that would do this)?

Cheers,
John


$ more VERSION
eprints-3.3.7
EPrints 3.3.7 (Bilberry) [Born on 2011-12-09]

$ bin/import digitallibrary  --verbose --enable-file-imports --enable-web-imports --user 1 archive XML /tmp/files/test.xml

Starting EPrints Repository.
Connecting to DB ... done.
Use of uninitialized value in string eq at /opt/eprints3/bin/../perl_lib/EPrints/DataObj/Document.pm line 308.
Use of uninitialized value in string eq at /opt/eprints3/bin/../perl_lib/EPrints/DataObj/Document.pm line 308.
Use of uninitialized value $filename in pattern match (m//) at (eval 440) line 145.
Use of uninitialized value $filename in pattern match (m//) at (eval 440) line 15.
Use of uninitialized value $filename in pattern match (m//) at (eval 440) line 19.
Use of uninitialized value $filename in pattern match (m//) at (eval 440) line 23.
Use of uninitialized value $filename in pattern match (m//) at (eval 440) line 27.
Use of uninitialized value $filename in substitution (s///) at /opt/eprints3/perl_lib/EPrints/Plugin/Storage/Local.pm line 249.
Use of uninitialized value $filename in substitution (s///) at /opt/eprints3/perl_lib/EPrints/Plugin/Storage/Local.pm line 250.
Use of uninitialized value $filename in substitution (s///) at /opt/eprints3/perl_lib/EPrints/Plugin/Storage/Local.pm line 251.
Use of uninitialized value $fn in concatenation (.) or string at /opt/eprints3/perl_lib/EPrints/Plugin/Storage/Local.pm line 49.
Use of uninitialized value $fn in concatenation (.) or string at /opt/eprints3/perl_lib/EPrints/Plugin/Storage/Local.pm line 55.
Use of uninitialized value $fn in concatenation (.) or string at /opt/eprints3/perl_lib/EPrints/Plugin/Storage/Local.pm line 57.
Unable to write to /opt/eprints3/archives/digitallibrary/documents/disk0/00/00/00/42/01/: Is a directory
EPrints::Storage: No plugins available to store file/361
Use of uninitialized value in concatenation (.) or string at /opt/eprints3/bin/../perl_lib/EPrints/DataObj/File.pm line 764.
file/361::set_file() failed: No storage plugins succeeded
Failed to create subdataobj on document.files at /opt/eprints3/bin/../perl_lib/EPrints/DataSet.pm line 1008
Failed to create subdataobj on archive.documents at /opt/eprints3/bin/../perl_lib/EPrints/DataSet.pm line 1008
Ending EPrints Repository.

-----Original Message-----
From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of sf2
Sent: 21 June 2012 19:34
To: eprints-tech@ecs.soton.ac.uk
Subject: [EP-tech] Re: Importing files from another system

Hi John,

You can safely ignore all the "id" fields as the import script will
override them. I think you're right in saying that EPrints will set the
mime type by itself. I'm not too sure whether you need to include the
"main" and "filename" :-/

I've added your XML to the wiki page if that's ok with you
(http://wiki.eprints.org/w/Import_Files)!

Seb.

On Thu, 21 Jun 2012 18:07:13 +0100, John Salter <J.Salter@leeds.ac.uk>
wrote:
> Hi Seb,
> Thanks for this - but how much of the data structure should I include
when
> trying to make a new eprint?
> E.g. I don't have an eprintid, docid, fileid, so can't include them.
> 
> I /could/ supply a mime_type, but should eprints be able to work this out
> from the file?
> Should the filename be calculated from the <url>, or do I have to specify
> this? (I see some fun with XSLT coming my way :o)
> 
>>From a bit more testing, I think this is the minimal set of data to get a
>>file into the system, from the local file system (although you'd want to
>>map more data in there if you had it!):
> 
> <?xml version='1.0' encoding='utf-8'?>
> <eprints xmlns='http://eprints.org/ep2/data/2.0'>
>   <eprint>
>     <documents>
>       <document>
>         <files>
>           <file>
>             <datasetid>document</datasetid>
>             <filename>thumb.jpg</filename>
>             <url>file:///tmp/files/thumb.jpg</url>
>           </file>
>         </files>
>         <format>image</format>
>         <security>public</security>
>         <main>thumb.jpg</main>
>       </document>
>     </documents>
>     <eprint_status>archive</eprint_status>
>     <userid>1</userid>
>     <type>image</type>
>     <creators>
>       <item>
>         <name>
>           <family>test</family>
>           <given>test</given>
>         </name>
>         <id>test@test.com</id>
>       </item>
>     </creators>
>     <title>test</title>
>   </eprint>
> </eprints>
> 
> Hope that helps someone in the future - and thanks for your help today!
> John
> 
> -----Original Message-----
> From: eprints-tech-bounces@ecs.soton.ac.uk
> [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of Sebastien
> Francois
> Sent: 21 June 2012 15:08
> To: eprints-tech@ecs.soton.ac.uk
> Subject: [EP-tech] Re: Importing files from another system
> 
> Hi John,
> 
> See an example attached - tested on 3.3.9, imported this way:
> 
> bin/import <REPO_ID> --enable-file-imports --enable-web-imports --user 1 
> --verbose eprint XML ./export2.xml
> 
> Seb.
> 
> PS: in the attached XML, I replaced the URL of my dev machine with 
> __REPO_URL__, change this to your repository URL.
> 
> On 21/06/12 14:55, John Salter wrote:
>> Yep - that didn't work either :o\
>>
>> According to this:
>> http://wiki.eprints.org/w/Import_From_URL
>> I was also missing a<files>  around the<file>.
>> I've added that and re-tried all the options, so far none work :o\
>>
>> Any other suggestions?
>>
>> Cheers,
>> John
>>
>> -----Original Message-----
>> From: eprints-tech-bounces@ecs.soton.ac.uk
>> [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of Jose Martin
>> Sent: 21 June 2012 14:40
>> To: eprints-tech@ecs.soton.ac.uk
>> Subject: [EP-tech] Re: Importing files from another system
>>
>> Have you tried for local files
>>
>> 	<url>file:///path_to_file</url>
>>
>> ?
>>
>> -----Original Message-----
>> From: eprints-tech-bounces@ecs.soton.ac.uk
>> [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of John Salter
>> Sent: 21 June 2012 14:24
>> To: 'eprints-tech@ecs.soton.ac.uk'
>> Subject: [EP-tech] Re: Importing files from another system
>>
>> Nothing silly in the verbose output.
>>
>> I can wget the file over http (when trying the http method), but even
>> using the file from the local filesystem doesn't work :o\
>>
>>
>> -----Original Message-----
>> From: eprints-tech-bounces@ecs.soton.ac.uk
>> [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of Yuri
>> Sent: 21 June 2012 13:48
>> To: eprints-tech@ecs.soton.ac.uk
>> Subject: [EP-tech] Re: Importing files from another system
>>
>> run the command with --verbose --verbose and try to see what happen. Can
>> you go to port 80 from the server to another server? Check your firewall
>> settings.
>>
>> Il 21/06/2012 14:22, John Salter ha scritto:
>>> Hi,
>>> I'm trying to import files from another system, into EPrints.
>>> I can either copy the files onto the server, or make them available
over
>>> http.
>>>
>>> In the XML, I've tried specifying (in an<eprints><eprint>   node)
>>> <document>
>>> 	<file href="/path/to/my.pdf" />
>>> </document>
>>>    - or -
>>> <document>
>>> 	<file>
>>> 		<url>http://url.of/my.pdf</url>
>>> 	</file>
>>> </document>
>>>
>>> And importing with:
>>>> bin/import ARCHIVENAME --enable-file-imports --enable-web-imports
>>>> --user 1 archive XML test.xml
>>> Both import the metadata from the XML OK, but neither option attaches
>>> the file to the eprint.
>>>
>>> What am I doing wrong - or will this even work?
>>>
>>> Cheers,
>>> John
>>> *** Options:
>>> http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>>> *** Archive: http://www.eprints.org/tech.php/
>>> *** EPrints community wiki: http://wiki.eprints.org/
>> *** Options:
http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
>>
>> *** Options:
http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
>>
>> *** Options:
http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
>>
>> *** Options:
http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
> 
> 
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: http://www.eprints.org/tech.php/
> *** EPrints community wiki: http://wiki.eprints.org/
*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/