[GMG-Devel] debugging a large file upload, EOFError

[Alon Levy]

Hi,

 I see there are some tickets about large file suport (647, 834, 419)
but I'm not sure if I'm hitting a limitation of gmg or of my webserver
or the fcgi or firefox perhaps, so I thought I'd give details of my
setup and ask for help:

 Submitting a 1.4GB video file from firefox 33.1 to mediagoblin running
in the same network (wifi though), firefox shows a message about timeout
(sadly I closed the window) and in mediagoblin I get an EOFError
exception, details below.

 I'm using nginx with fastcgi. I think I got a 499: (just looking now at
the logs)

10.81.2.1 - - [21/Nov/2014:20:17:27 +0200] "POST /submit/ HTTP/1.1" 499
0 "http://mediagoblin.tami.org.il/submit/"; "Mozilla/5.0 (X11; Linux
x86_64; rv:33.0) Gecko/20100101 Firefox/33.0" "10.81.2.231"

 nginx.conf has 'client_max_body_size 2560m;'


 Does anyone have a working setup for two gigabyte uploads? If so could
you describe it? Any ideas why I cannot upload large files? I don't know
what the cutoff is, I guess that would be a next step. I hope someone
has a better suggestion,

Alon
p.s. complete stack trace from mediagoblin.log:

Error - <type 'exceptions.EOFError'>:
URL: http://mediagoblin.tami.org.il/submit/
File
'/usr/lib/python2.7/site-packages/paste/exceptions/errormiddleware.py',
line 144 in __call__
  app_iter = self.application(environ, sr_checker)
File '/home/mediagoblin/mediagoblin/mediagoblin/app.py', line 268 in
__call__
  return self.call_backend(environ, start_response)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wsgi.py',
line 411 in __call__
  return self.app(environ, start_response)
File '/home/mediagoblin/mediagoblin/mediagoblin/app.py', line 188 in
call_backend
  request.locale = translate.get_locale_from_request(request)
File '/home/mediagoblin/mediagoblin/mediagoblin/tools/translate.py',
line 97 in get_locale_from_request
  request_args = (request.args, request.form)[request.method=='POST']
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/utils.py',
line 77 in __get__
  value = self.func(obj)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wrappers.py',
line 383 in form
  self._load_form_data()
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wrappers.py',
line 318 in _load_form_data
  data = parser.parse_from_environ(self.environ)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/formparser.py',
line 164 in parse_from_environ
  return self.parse(stream, mimetype, content_length, options)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/formparser.py',
line 188 in parse
  content_length, options)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/formparser.py',
line 100 in wrapper
  stream.exhaust()
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wsgi.py',
line 787 in exhaust
  self.read(chunk)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wsgi.py',
line 801 in read
  read = self._read(to_read)
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
line 161 in read
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
line 147 in _waitForData
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
line 696 in process_input
File
'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
line 474 in read
EOFError:


CGI Variables
-------------
  CONTENT_LENGTH: '1500501143'
  CONTENT_TYPE: 'multipart/form-data;
boundary=---------------------------4982243468044615281985221407'
  DOCUMENT_ROOT: '/usr/share/nginx/html'
  DOCUMENT_URI: '/submit/'
  GATEWAY_INTERFACE: 'CGI/1.1'
  HTTP_ACCEPT:
'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
  HTTP_ACCEPT_ENCODING: 'gzip, deflate'
  HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
  HTTP_CONTENT_LENGTH: '1500501143'
  HTTP_CONTENT_TYPE: 'multipart/form-data;
boundary=---------------------------4982243468044615281985221407'
  HTTP_COOKIE: 'mediagoblin_csrftoken=2178534525516131666;
MGSession=eyJzdGF5X2xvZ2dlZF9pbiI6dHJ1ZSwibWVzc2FnZXMiOltdLCJ1c2VyX2lkIjoiMSJ9.B04tgg.s1z0ifQ23U1HlKy6Z-6D1Kb7Mjk;
mediagoblin_csrftoken=2178534525516131666'
  HTTP_HOST: 'mediagoblin.tami.org.il'
  HTTP_REFERER: 'http://mediagoblin.tami.org.il/submit/'
  HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64; rv:33.0)
Gecko/20100101 Firefox/33.0'
  HTTP_X_FORWARDED_FOR: '10.81.2.231'
  PATH_INFO: '/submit/'
  REDIRECT_STATUS: '200'
  REMOTE_ADDR: '10.81.2.1'
  REMOTE_PORT: '54761'
  REQUEST_METHOD: 'POST'
  REQUEST_URI: '/submit/'
  SERVER_ADDR: '10.81.2.247'
  SERVER_NAME: 'localhost'
  SERVER_PORT: '80'
  SERVER_PROTOCOL: 'HTTP/1.1'
  SERVER_SOFTWARE: 'nginx/1.4.7'


WSGI Variables
--------------
  application: <mediagoblin.app.MediaGoblinApp object at 0x21a0450>
  paste.throw_errors: True
  werkzeug.request: <Request 'http://mediagoblin.tami.org.il/submit/'
[POST]>
  wsgi process: 'Multiprocess'