#1756 out of memory issues with 1.19.0
Closed: Duplicate 4 years ago by tkopecek. Opened 4 years ago by kevin.

Greetings. We are seeing some odd out of memory issues after upgrading koji.stg.fedoraproject.org to 1.19.0. ;(

[Tue Nov 05 18:04:10.506107 2019] [:error] [pid 25244] 2019-11-05 18:04:10,506 [WARNING] m=listTagg
ed u=None p=25244 r=10.5.128.177:53202 koji.xmlrpc: Traceback (most recent call last):
[Tue Nov 05 18:04:10.506142 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojixmlrpc.py", 
line 235, in _wrap_handler
[Tue Nov 05 18:04:10.506146 2019] [:error] [pid 25244]     response = handler(environ)
[Tue Nov 05 18:04:10.506148 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojixmlrpc.py", 
line 278, in handle_rpc
[Tue Nov 05 18:04:10.506164 2019] [:error] [pid 25244]     return self._dispatch(method, params)
[Tue Nov 05 18:04:10.506166 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojixmlrpc.py", 
line 315, in _dispatch
[Tue Nov 05 18:04:10.506168 2019] [:error] [pid 25244]     ret = koji.util.call_with_argcheck(func,
 params, opts)
[Tue Nov 05 18:04:10.506171 2019] [:error] [pid 25244]   File "/usr/lib/python2.7/site-packages/koj
i/util.py", line 261, in call_with_argcheck
[Tue Nov 05 18:04:10.506173 2019] [:error] [pid 25244]     return func(*args, **kwargs)
[Tue Nov 05 18:04:10.506175 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojihub.py", lin
e 10487, in listTagged
[Tue Nov 05 18:04:10.506177 2019] [:error] [pid 25244]     results = readTaggedBuilds(tag, event, i
nherit=inherit, latest=latest, package=package, owner=owner, type=type)
[Tue Nov 05 18:04:10.506192 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojihub.py", lin
e 1269, in readTaggedBuilds
[Tue Nov 05 18:04:10.506194 2019] [:error] [pid 25244]     packages = readPackageList(tagID=tag, ev
ent=event, inherit=True, pkgID=package)
[Tue Nov 05 18:04:10.506196 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojihub.py", lin
e 1135, in readPackageList
[Tue Nov 05 18:04:10.506198 2019] [:error] [pid 25244]     for p in _multiRow(q, locals(), [pair[1]
 for pair in fields]):
[Tue Nov 05 18:04:10.506200 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojihub.py", lin
e 4903, in _multiRow
[Tue Nov 05 18:04:10.506203 2019] [:error] [pid 25244]     return [dict(zip(fields, row)) for row i
n _fetchMulti(query, values)]
[Tue Nov 05 18:04:10.506205 2019] [:error] [pid 25244]   File "/usr/share/koji-hub/kojihub.py", lin
e 4875, in _fetchMulti
[Tue Nov 05 18:04:10.506221 2019] [:error] [pid 25244]     c.execute(query, values)
[Tue Nov 05 18:04:10.506223 2019] [:error] [pid 25244]   File "/usr/lib/python2.7/site-packages/koj
i/db.py", line 136, in execute
[Tue Nov 05 18:04:10.506225 2019] [:error] [pid 25244]     ret = self.cursor.execute(operation, par
ameters)
[Tue Nov 05 18:04:10.506241 2019] [:error] [pid 25244] DatabaseError: out of memory for query resul
t
[Tue Nov 05 18:04:04.475066 2019] [:error] [pid 25243] 474 [ERROR] m=taginfo u=None p=25243 r=10.5.
128.177:49046 koji.web: Traceback (most recent call last):
[Tue Nov 05 18:04:04.475093 2019] [:error] [pid 25243]   File "/usr/share/koji-web/scripts/wsgi_pub
lisher.py", line 372, in handle_request
[Tue Nov 05 18:04:04.475096 2019] [:error] [pid 25243]     result = func(environ, **data)
[Tue Nov 05 18:04:04.475098 2019] [:error] [pid 25243]   File "/usr/share/koji-web/scripts/index.py
", line 890, in taginfo
[Tue Nov 05 18:04:04.475101 2019] [:error] [pid 25243]     numPackages = server.count('listPackages
', tagID=tag['id'], inherited=True)
[Tue Nov 05 18:04:04.475103 2019] [:error] [pid 25243]   File "/usr/lib/python2.7/site-packages/koj
i/__init__.py", line 2135, in __call__
[Tue Nov 05 18:04:04.475105 2019] [:error] [pid 25243]     return self.__func(self.__name, args, op
ts)
[Tue Nov 05 18:04:04.475107 2019] [:error] [pid 25243]   File "/usr/lib/python2.7/site-packages/koj
i/__init__.py", line 2650, in _callMethod
[Tue Nov 05 18:04:04.475110 2019] [:error] [pid 25243]     raise err
[Tue Nov 05 18:04:04.475112 2019] [:error] [pid 25243] Fault: <Fault 1: "<class 'psycopg2.DatabaseE
rror'>: out of memory for query result\\n">
[Tue Nov 05 18:04:04.475114 2019] [:error] [pid 25243] 
[Tue Nov 05 18:04:04.463135 2019] [:error] [pid 25036]     SELECT package.id, package.name, tag.id,
 tag.name, users.id, users.name, extra_arches, tag_packages.blocked
[Tue Nov 05 18:04:04.463140 2019] [:error] [pid 25036]     FROM tag_packages
[Tue Nov 05 18:04:04.463142 2019] [:error] [pid 25036]     JOIN tag on tag.id = tag_packages.tag_id
[Tue Nov 05 18:04:04.463144 2019] [:error] [pid 25036]     JOIN package ON package.id = tag_package
s.package_id
[Tue Nov 05 18:04:04.463159 2019] [:error] [pid 25036]     JOIN tag_package_owners ON
[Tue Nov 05 18:04:04.463161 2019] [:error] [pid 25036]         tag_packages.tag_id = tag_package_ow
ners.tag_id AND
[Tue Nov 05 18:04:04.463164 2019] [:error] [pid 25036]         tag_packages.package_id = tag_packag
es.package_id
[Tue Nov 05 18:04:04.463166 2019] [:error] [pid 25036]     JOIN users ON users.id = tag_package_own
ers.owner
[Tue Nov 05 18:04:04.463168 2019] [:error] [pid 25036]     WHERE (tag_packages.active = TRUE) AND (
tag_package_owners.active = TRUE)
[Tue Nov 05 18:04:04.463198 2019] [:error] [pid 25036]         AND tag.id = 412
[Tue Nov 05 18:04:04.469585 2019] [:error] [pid 25036] 2019-11-05 18:04:04,469 [WARNING] m=count u=
None p=25036 r=10.5.128.177:49048 koji.xmlrpc: Traceback (most recent call last):
[Tue Nov 05 18:04:04.469643 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojixmlrpc.py", 
line 235, in _wrap_handler
[Tue Nov 05 18:04:04.469648 2019] [:error] [pid 25036]     response = handler(environ)
[Tue Nov 05 18:04:04.469656 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojixmlrpc.py", 
line 278, in handle_rpc
[Tue Nov 05 18:04:04.469659 2019] [:error] [pid 25036]     return self._dispatch(method, params)
[Tue Nov 05 18:04:04.469661 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojixmlrpc.py", 
line 315, in _dispatch
[Tue Nov 05 18:04:04.469663 2019] [:error] [pid 25036]     ret = koji.util.call_with_argcheck(func,
 params, opts)
[Tue Nov 05 18:04:04.469666 2019] [:error] [pid 25036]   File "/usr/lib/python2.7/site-packages/koj
i/util.py", line 261, in call_with_argcheck
[Tue Nov 05 18:04:04.469670 2019] [:error] [pid 25036]     return func(*args, **kwargs)
[Tue Nov 05 18:04:04.469710 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojihub.py", lin
e 11845, in count
[Tue Nov 05 18:04:04.469714 2019] [:error] [pid 25036]     result = getattr(self, methodName)(*args
, **kw)
[Tue Nov 05 18:04:04.469717 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojihub.py", lin
e 11062, in listPackages
[Tue Nov 05 18:04:04.469719 2019] [:error] [pid 25036]     event=event).values())
[Tue Nov 05 18:04:04.469722 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojihub.py", lin
e 1161, in readPackageList
[Tue Nov 05 18:04:04.469725 2019] [:error] [pid 25036]     for p in _multiRow(q, locals(), [pair[1]
 for pair in fields]):
[Tue Nov 05 18:04:04.469727 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojihub.py", lin
e 4903, in _multiRow
[Tue Nov 05 18:04:04.469730 2019] [:error] [pid 25036]     return [dict(zip(fields, row)) for row i
n _fetchMulti(query, values)]
[Tue Nov 05 18:04:04.469733 2019] [:error] [pid 25036]   File "/usr/share/koji-hub/kojihub.py", lin
e 4875, in _fetchMulti
[Tue Nov 05 18:04:04.469736 2019] [:error] [pid 25036]     c.execute(query, values)
[Tue Nov 05 18:04:04.469739 2019] [:error] [pid 25036]   File "/usr/lib/python2.7/site-packages/koj
i/db.py", line 136, in execute
[Tue Nov 05 18:04:04.469741 2019] [:error] [pid 25036]     ret = self.cursor.execute(operation, par
ameters)
[Tue Nov 05 18:04:04.469744 2019] [:error] [pid 25036] DatabaseError: out of memory for query resul
t

This could well be something messed up in our setup, but I am not sure what.

Any ideas?


We've a bug in that query (found in #1752), Can you try PR #1753 it is a proper fix for you?

Metadata Update from @tkopecek:
- Custom field Size adjusted to None

4 years ago

Yep. This did indeed seem to fix it. :)

Feel free to close this....

Perhaps we could get a 1.19.1 soon?

Duplicate of #1752 - and yes, we're planning 1.19.1 in a ~week.

Metadata Update from @tkopecek:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata