EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #07491

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] EPrints upload error on files over ~130k

For anyone who's interested in this thread in the future: alterations to the SecRequestBody part of the ModSec ruleset fixed the problem. I'll add more details when (and if) central ITS provides them.

Cheers,
Kelly

Kelly Phillips
Archivist - Digital Programs
Special Collections and Archives
Cline Library
Northern Arizona University
928-523-5038

-----Original Message-----
From: eprints-tech-bounces@ecs.soton.ac.uk <eprints-tech-bounces@ecs.soton.ac.uk> On Behalf Of David R Newman
Sent: Wednesday, August 29, 2018 2:25 PM
To: eprints-tech@ecs.soton.ac.uk
Subject: Re: [EP-tech] EPrints upload error on files over ~130k

Hi Kelly,

So I suspected that some patching may have been done.  That in itself may have created the issue and/or a reboot subsequent to patching.  
However, I would have thought you would have this server had patched and rebooted a number of times in the past without issue.  Yuri's suggestion SELinux may have been enabled is a possiblity, if your central ITS was doing a security review and saw this was disabled.  EPrints will work with SELinux enabled but there are quite a few complex changes that would need to be made to ensure all functionality works, in particular file upload.  If it was just enabled without any testing, you would likely have problems.

ModSecurity is not an Apache module required by EPrints but maybe it is something your central ITS wanted to ensure some specific security requirement.  It looks like ModSecurity is a firewall implemented at Apache level.  As well as restricting IPs and ports, it presumably can restrict on hostname (of a website) and I can see it can also limit max upload size.  I think the option you need to look for in your Apache configuration is SecRequestBodyLimit.  Looking at https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSpiderLabs%2FModSecurity%2Fwiki%2FReference-Manual-%2528v2.x%2529%23SecRequestBodyLimit&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=LADOv9z5xJnMc3S%2BN9nzDxAuLO5OGqQ%2FQ0hsU4K3uoU%3D&amp;reserved=0
the default limit seems to be be 130MB rather than KB but that seems too be much of a coincidence to your 130KB.

Regards

David Newman


On 29/08/2018 21:29, Kelly Kathleen Phillips wrote:
> Thanks, David and Yuri, for your responses.
>
> The problem seems to have arisen after general server patching performed by our central ITS. I did test uploads after the patch, but it's possible I used a very small file to do so, and wouldn't have seen the error.
>
> Yuri, the 2014 fix for an error that expressed itself the same way is described here: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints%2Fissues%2F287&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=s5yy9%2F5CVLR5PgLmqbmf3iEo2OOq4xZZhyQdZFKpikw%3D&amp;reserved=0 and could have had a different cause. In our case, my best guess is that the JSON snippets fed to the javascript progress bar routine aren't being generated and delivered properly - if I'm understanding that right, then I'm wondering if the same error could be caused by the server refusing an upload outside of EPrints.
>
> Nothing about the EPrints configuration has changed since long before the patching happened, so I'd be surprised if it had to do with the temp directory - unless a security patch has cut off access to that location. Seems like there's a lot of ModSecurity action in the ssl_error_log that I don't remember seeing before the patching. I don't feel confident in my reading of the log or free to disable any server security features on my own, so I've got a call out to ITS. I think I will try David's solution on our dev server, however, just to see what happens.
>
> Kelly Phillips
> Archivist - Digital Programs
> Special Collections and Archives
> Cline Library
> Northern Arizona University
> 928-523-5038
>
> -----Original Message-----
> From: eprints-tech-bounces@ecs.soton.ac.uk 
> <eprints-tech-bounces@ecs.soton.ac.uk> On Behalf Of Yuri
> Sent: Tuesday, August 28, 2018 11:30 PM
> To: eprints-tech@ecs.soton.ac.uk
> Subject: Re: [EP-tech] EPrints upload error on files over ~130k
>
> I would disable SELinux temporary to see if it is the guilty. What was the fix in 2014?
>
>
> Il 29/08/2018 00:57, David R Newman ha scritto:
>> Hi Kelly,
>>
>> Sorry, I misread your subject line as "upload error on file of ~130k"
>> for some reason.  I do no believe that there would be any restriction 
>> prevent files over 130k being uploaded.  The default limit is 1GB.
>> This may still be the tmp directory issue I described previously, as 
>> sufficiently small files I suspect will be written directly to the 
>> correct place on the filesystem, whereas anything over this size will 
>> be transferring the file piecemeal and therefore will collect in the 
>> tmp directory and only be moved once the upload is complete.
>>
>> Regards
>>
>> David Newman
>>
>>
>> On 28/08/2018 22:52, David R Newman wrote:
>>> Hi Kelly,
>>>
>>> This could be one of a number of issues.  The first thing I should 
>>> check with you is if this is a one off issue with a particular file 
>>> or EPrint record or if it is affecting you uploading any files to 
>>> your repository?
>>>
>>> My best guess on the issue, would be that you either need to 
>>> configure EPrints temporary directory or ensure that the temporary 
>>> directory you have specified exists and is writeable to by the 
>>> webserver (Apache).  (However, I cannot see why this would have 
>>> suddenly changed if it was working before). Typically, I would add 
>>> the tmp directory configuration to your archive's session.pl (e.g.
>>> /opt/eprints3/archives/nau/cfg/cfg.d/session.pl) in session_init and 
>>> reload Apache.  Something like:
>>>
>>> $c->{session_init} = sub
>>> {
>>>          my( $repository, $offline ) = @_;
>>>
>>>          $ENV{'USER'} = 'eprints';
>>>          $ENV{'HOME'} = '/home/eprints';
>>>          if ( -d "/opt/eprints3/tmp/" )
>>>          {
>>>                  $ENV{'TMPDIR'} = '/opt/eprints3/tmp/';
>>>          }
>>> };
>>>
>>> The reason I suspect it is this issue, is because RHEL 7 will not by 
>>> default allow EPrints to write to /tmp/ if the Apache user is set to 
>>> eprints in /etc/httpd/conf/httpd.conf, as is generally recommended.
>>> Therefore, you need to specify EPrints' own tmp directory, the above 
>>> example uses /opt/eprints3/tmp/.  If you have this configuration in 
>>> place, you should also make sure that /opt/eprints3/tmp/ has the 
>>> following permissions and user/group ownership (using "ls -la 
>>> /opt/eprints3/tmp/" and checking the first line of the output):
>>>
>>> drwxrwsr-x 12 eprints eprints    77824 Aug 28 21:35 .
>>>
>>> Even if this is the same as above (bar the size of the directory and 
>>> the modified date), I would further recommend clearing out the 
>>> /opt/eprints3/tmp/ as it will not be cleared on reboot like /tmp/ 
>>> and may be using up unnecessary space. Typically tasks clear up 
>>> stuff they have put in EPrints' tmp directory but over time there 
>>> will be the odd thing that does not get cleared and over months and 
>>> years this may start to waste a significant amount space.
>>>
>>> Regards
>>>
>>> David Newman
>>>
>>>
>>> On 28/08/2018 20:22, Kelly Kathleen Phillips wrote:
>>>> Hi,
>>>>
>>>> We're currently having a problem uploading files over about 130 kb 
>>>> to our institutional repository. We're running EPrints 3.3.15 on 
>>>> Red Hat EL 7.5.
>>>>
>>>> We get this error from the Quick Upload tool on the user homepage:
>>>>
>>>> "Internal Server Error
>>>>
>>>> The server encountered and internal error or misconfiguration and 
>>>> was unable to complete your request.
>>>>
>>>> Please contact the server administrator at root@localhost to inform 
>>>> them of the time this error occurred, and the actions you performed 
>>>> just before this error.
>>>>
>>>> More information about this error may be available in the server 
>>>> error log."
>>>>
>>>> We get this error on the "Edit Item: Add a new document" screen:
>>>>
>>>> "Request for
>>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopenknowledge.nau.edu%2Fcgi%2Fusers%2Fajax%2Fupload_progress%3Fprogre&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=IKH12%2Bgx8I8esEJDGKqnDYyY3JZE9uzALrscnXfQfbI%3D&amp;reserved=0
>>>> ssid=2B24D4FBFE9742949458083C6D5C3228
>>>> failed: 404 Not Found"
>>>>
>>>> Something similar to the second error has cropped up in 2014, and 
>>>> Jidai Yao addressed it, but that fix is already present in our code.
>>>>
>>>> I'm early in the troubleshooting process (just getting to the 
>>>> server logs etc.) but if anyone has run across this and/or has any 
>>>> suggestions, I would be most grateful - we've just been advertising 
>>>> this service to our new faculty for the year, we'd rather not this 
>>>> be their first experience of it.
>>>>
>>>> Kelly Phillips
>>>>
>>>> Archivist - Digital Programs
>>>>
>>>> *Special Collections and Archives*
>>>>
>>>> *Cline Library*
>>>>
>>>> *Northern Arizona University*
>>>>
>>>> 928-523-5038
>>>>
>>>>
>>>>
>>>> *** 
>>>> Options:http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tec
>>>> h
>>>> *** Archive:http://www.eprints.org/tech.php/
>>>> *** EPrints community wiki:http://wiki.eprints.org/
>>>> *** EPrints developers Forum:http://forum.eprints.org/
>>>
>>>
>>> *** 
>>> Options:http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>>> *** Archive:http://www.eprints.org/tech.php/
>>> *** EPrints community wiki:http://wiki.eprints.org/
>>> *** EPrints developers Forum:http://forum.eprints.org/
>>
>>
>> *** Options: 
>> http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=s6LN%2B%2FqkiglPNOOyRpbpqtFbD8PdqHUtV9djS55Pw7E%3D&amp;reserved=0
>> *** EPrints community wiki: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=LkTBBq%2F%2BIx1T%2FP2hdUZCFps8hQDbRGFDX4YQZ%2By%2Fh%2F8%3D&amp;reserved=0
>> *** EPrints developers Forum: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=psT4kLfxtl5x0L85hBl22wqCKfoHIK7hViinYVqg2xc%3D&amp;reserved=0
> *** Options: 
> http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=s6LN%2B%2FqkiglPNOOyRpbpqtFbD8PdqHUtV9djS55Pw7E%3D&amp;reserved=0
> *** EPrints community wiki: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=LkTBBq%2F%2BIx1T%2FP2hdUZCFps8hQDbRGFDX4YQZ%2By%2Fh%2F8%3D&amp;reserved=0
> *** EPrints developers Forum: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=psT4kLfxtl5x0L85hBl22wqCKfoHIK7hViinYVqg2xc%3D&amp;reserved=0
>
> *** Options: 
> http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=s6LN%2B%2FqkiglPNOOyRpbpqtFbD8PdqHUtV9djS55Pw7E%3D&amp;reserved=0
> *** EPrints community wiki: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=LkTBBq%2F%2BIx1T%2FP2hdUZCFps8hQDbRGFDX4YQZ%2By%2Fh%2F8%3D&amp;reserved=0
> *** EPrints developers Forum: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=psT4kLfxtl5x0L85hBl22wqCKfoHIK7hViinYVqg2xc%3D&amp;reserved=0

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=s6LN%2B%2FqkiglPNOOyRpbpqtFbD8PdqHUtV9djS55Pw7E%3D&amp;reserved=0
*** EPrints community wiki: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=LkTBBq%2F%2BIx1T%2FP2hdUZCFps8hQDbRGFDX4YQZ%2By%2Fh%2F8%3D&amp;reserved=0
*** EPrints developers Forum: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9f240af8d2fc4a1e182b08d61e82c409%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=psT4kLfxtl5x0L85hBl22wqCKfoHIK7hViinYVqg2xc%3D&amp;reserved=0