EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #08775

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Datafiles size limit of 10 GB

Hi Angeliki,


I would normally advise against using EPrints for files larger than 10GB, as the time to upload can be quite long and as they is no in-built failure recovery, there is a higher chance that uploads may fail and it will ultimately take longer to upload than the same amount of data in smaller files.  However, if the data is being uploaded from fairly locally over a high bandwidth connections you may be OK.


The _javascript_ that manages uploads can configured to limit the max upload size but this is not the case by default.  However, if you check 88_uploadmethod_file.js and there is code like:


checkFilesize: function(file){
        if (file.size < max_size)
        {
            return 1;
        }
        eprints.currentRepository().phrase( {'Plugin/Screen/EPrint/UploadMethod/File:filesize_too_big':{} },function(phrase){ alert (file.name+": \n"+ phrase["Plugin/Screen/EPrint/UploadMethod/File:filesize_too_big"]);  } ) ;
        return 0;
    },


and max_size is set somewhere else in the script.  Then you can modify the value for max_size to allow for bigger files.  If there is no max_size set then you should be OK to upload any file size.  However, be sure to edit the right version of 88_uploadmethod_file.js as this can appear on lib/, site_lib/ and archive level subdirectories.  You want to edit the one at archive level.  If it does not exist copy the one from site_lib or if that does not exist lib and then edit it at the archive level (archives/ARCHIVE_NAME/cfg/static/_javascript_/auto/88_uploadmethod_file.js).


I don't know if web browsers have built in upload limits anymore but if they do I would not be surprised if 10GB was a limit they may have in place.  So if you still cannot upload large files it may be worth investigating if it is due to your web browser.  However, if the upload limit is not restricted by the setting of max_size, I would not be surprised if the upload did fail at some point because such a large file has more opporunity due to network and other issues.  There is also a small possibility that 10GB leads to some integer overflow somewhere.  It may be in the _javascript_ in 88_uploadmethod_file.js but I am not aware of this being an issue but I have not tried to upload files over 2-3GB.


Regards


David Newman


On 18/10/2021 09:58, AndrikAZ--- via Eprints-tech wrote:
CAUTION: This e-mail originated outside the University of Southampton.

Hello All,


In our Ubira eData repository, we have a limit on the uploaded datasets of 10 GB per record. There are cases though, with much larger datasets that have to be deposited in a single record. Would it be possible to exceed this 10 GB limit? 


Any recommendation will be very appreciated.


Many thanks,

Angeliki



Angeliki Andrikopoulou

Research Data Librarian

 

Library Services

Skype: A.Andrikopoulou@bham.ac.uk

e: A.Andrikopoulou@bham.ac.uk

https://intranet.birmingham.ac.uk/library



*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/

Virus-free. www.avg.com