#5231 Storing big file as a secret in vault raises traceback
Closed: Fixed None Opened 8 years ago by alich.

VERSION: 4.2.90.201508141119GITfe91c37, API_VERSION: 2.148

-sh-4.3$ dd if=/dev/zero of=huge-file bs=1024k count=100; ipa vault-archive SharedG3 --shared --in=huge-file; rm -rf huge-file
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.219098 s, 479 MB/s
ipa: ERROR: non-public: MemoryError: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 130, in execute
    result = self.Command[_name](*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
    ret = self.run(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1109, in run
    return self.forward(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1091, in forward
    return self.api.Command.vault_archive_internal(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
    ret = self.run(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 761, in run
    return self.forward(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 782, in forward
    return self.Backend.rpcclient.forward(self.name, *args, **kw)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 950, in forward
    return self._call_command(command, params)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 927, in _call_command
    return command(*params)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1078, in _call
    return self.__request(name, args)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1047, in __request
    verbose=self.__verbose >= 3,
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1280, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 614, in single_request
    self.send_content(h, request_body)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 425, in send_content
    connection.endheaders(request_body)
  File "/usr/lib/python2.7/site-packages/ipapython/nsslib.py", line 291, in endheaders
    httplib.HTTPConnection.endheaders(self, message)
  File "/usr/lib64/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 891, in _send_output
    msg += message_body
MemoryError
ipa: ERROR: an internal error has occurred

VERSION: 4.2.90.201508141119GITfe91c37, API_VERSION: 2.148

-sh-4.3$ dd if=/dev/zero of=huge-file bs=1024k count=100; ipa vault-archive SharedG3 --shared --in=huge-file; rm -rf huge-file
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.219098 s, 479 MB/s
ipa: ERROR: non-public: MemoryError: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 130, in execute
    result = self.Command[_name](*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
    ret = self.run(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1109, in run
    return self.forward(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1091, in forward
    return self.api.Command.vault_archive_internal(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
    ret = self.run(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 761, in run
    return self.forward(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 782, in forward
    return self.Backend.rpcclient.forward(self.name, *args, **kw)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 950, in forward
    return self._call_command(command, params)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 927, in _call_command
    return command(*params)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1078, in _call
    return self.__request(name, args)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1047, in __request
    verbose=self.__verbose >= 3,
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1280, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 614, in single_request
    self.send_content(h, request_body)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 425, in send_content
    connection.endheaders(request_body)
  File "/usr/lib/python2.7/site-packages/ipapython/nsslib.py", line 291, in endheaders
    httplib.HTTPConnection.endheaders(self, message)
  File "/usr/lib64/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 891, in _send_output
    msg += message_body
MemoryError
ipa: ERROR: an internal error has occurred

For large files the plugin should use a temporary file instead of an in-memory buffer.

  • ab: we should handle exception properly, it is normal for WebApp to set limits. We can even set something to Apache config.
  • ab: Proposing limit of 10 MB

could be fixed only in client. E.g check file size and refuse to send anything bigger than 10MB.

master:

  • 02ab34c vault: Limit size of data stored in vault

ipa-4-2:

  • 9fc82bc vault: Limit size of data stored in vault

Metadata Update from @alich:
- Issue assigned to dkupka
- Issue set to the milestone: FreeIPA 4.2.1

7 years ago

Login to comment on this ticket.

Metadata