EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #08776

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Datafiles size limit of 10 GB

CAUTION: This e-mail originated outside the University of Southampton.

Hi Angeliki,

We have had successful uploads of files of up to 50GB in the past,
though we now impose an artificial limit of 30GB.

One thing to be aware of: in our experience at least, uploaded files
are saved into the server's /tmp directory before being copied into the
documents directory. So even if the documents directory is on a
separate drive with plenty of space, the upload will fail if there
isn't enough room left in /tmp. And by default the copy in /tmp will
persist until the next reboot, so several large uploads in a row can
cause problems.

Best wishes,
Alex.

On Mon, 2021-10-18 at 11:36 +0100, David R Newman via Eprints-tech
wrote:
> I don't know if web browsers have built in upload limits anymore but
> if they do I would not be surprised if 10GB was a limit they may have
> in place.  So if you still cannot upload large files it may be worth
> investigating if it is due to your web browser.  However, if the
> upload limit is not restricted by the setting of max_size, I would
> not be surprised if the upload did fail at some point because such a
> large file has more opporunity due to network and other issues.
> There is also a small possibility that 10GB leads to some integer
> overflow somewhere.  It may be in the Javascript in
> 88_uploadmethod_file.js but I am not aware of this being an issue but
> I have not tried to upload files over 2-3GB.
>
> On 18/10/2021 09:58, AndrikAZ--- via Eprints-tech wrote:
> > In our Ubira eData repository, we have a limit on the uploaded
> > datasets of 10 GB per record. There are cases though, with much
> > larger datasets that have to be deposited in a single record. Would
> > it be possible to exceed this 10 GB limit?
> >
> > Any recommendation will be very appreciated.

--
Alex Ball
Research Data Librarian (Systems)
University of Bath, Bath BA2 7AY, UK
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bath.ac.uk%2Flibrary%2F&amp;data=04%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C4d321aa028c4481c4ec208d992e84ba8%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637702348654631247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Ifzv0uV2mZpmrrh%2F57y5%2FTZLiNcL5Y8g9cfTfNojtI8%3D&amp;reserved=0