Issue #3366: PostgreSQL 14: KeyError: <class 'decimal.Decimal'>, TypeError: cannot marshal <class 'decimal.Decimal'> objects - koji

koji

#3366 PostgreSQL 14: KeyError: <class 'decimal.Decimal'>, TypeError: cannot marshal <class 'decimal.Decimal'> objects

Closed: Fixed 2 years ago by tkopecek. Opened 2 years ago by amessina.

After upgrading from Fedora 35 to Fedora 36 (koji-1.28.1-1.fc36.noarch, python3-3.10.4-1.fc36.x86_64), I now see the following xmlrpc.client.Fault for any connections to the hub (kojira, kojid, kojiweb):

[Sat May 21 15:00:50.947731 2022] [wsgi:error] 2022-05-21 15:00:50,947 [WARNING] m=getSessionInfo u=linux-ws1.messinet.com p=1090 r=fd00::1:46338 koji.xmlrpc: Traceback (most recent call last):
[Sat May 21 15:00:50.947794 2022] [wsgi:error]   File "/usr/lib64/python3.10/xmlrpc/client.py", line 522, in __dump
[Sat May 21 15:00:50.947807 2022] [wsgi:error]     f = self.dispatch[type(value)]
[Sat May 21 15:00:50.947825 2022] [wsgi:error] KeyError: <class 'decimal.Decimal'>
[Sat May 21 15:00:50.947843 2022] [wsgi:error] 
[Sat May 21 15:00:50.947851 2022] [wsgi:error] During handling of the above exception, another exception occurred:
[Sat May 21 15:00:50.947860 2022] [wsgi:error] 
[Sat May 21 15:00:50.947868 2022] [wsgi:error] Traceback (most recent call last):
[Sat May 21 15:00:50.947882 2022] [wsgi:error]   File "/usr/share/koji-hub/kojixmlrpc.py", line 257, in _wrap_handler
[Sat May 21 15:00:50.947893 2022] [wsgi:error]     response = dumps(response, methodresponse=1, marshaller=Marshaller)
[Sat May 21 15:00:50.947901 2022] [wsgi:error]   File "/usr/lib/python3.10/site-packages/koji/xmlrpcplus.py", line 83, in dumps
[Sat May 21 15:00:50.947910 2022] [wsgi:error]     data = m.dumps(params)
[Sat May 21 15:00:50.947919 2022] [wsgi:error]   File "/usr/lib64/python3.10/xmlrpc/client.py", line 514, in dumps
[Sat May 21 15:00:50.947928 2022] [wsgi:error]     dump(v, write)
[Sat May 21 15:00:50.947936 2022] [wsgi:error]   File "/usr/lib64/python3.10/xmlrpc/client.py", line 536, in __dump
[Sat May 21 15:00:50.947945 2022] [wsgi:error]     f(self, value, write)
[Sat May 21 15:00:50.947954 2022] [wsgi:error]   File "/usr/lib64/python3.10/xmlrpc/client.py", line 607, in dump_struct
[Sat May 21 15:00:50.947963 2022] [wsgi:error]     dump(v, write)
[Sat May 21 15:00:50.947972 2022] [wsgi:error]   File "/usr/lib64/python3.10/xmlrpc/client.py", line 526, in __dump
[Sat May 21 15:00:50.947981 2022] [wsgi:error]     raise TypeError("cannot marshal %s objects" % type(value))
[Sat May 21 15:00:50.947991 2022] [wsgi:error] TypeError: cannot marshal <class 'decimal.Decimal'> objects

I did not see this issue on Fedora 35. Any idea how to begin troubleshooting?

amessina commented 2 years ago

This may be related to the change in PostgreSQL 14 (https://www.postgresql.org/docs/14/release-14.html) "Change EXTRACT() to return type numeric instead of float8"

Temporarily, I have added the recommendation from https://www.psycopg.org/docs/faq.html#faq-float into /usr/lib/python3.10/site-packages/koji/db.py and my local Koji instance is running again.

Metadata Update from @tkopecek:
- Custom field Size adjusted to None
- Issue set to the milestone: 1.29.1

2 years ago

tkopecek commented 2 years ago

Second option is that we could live with Decimal internally and update xmlrpc ExtendedMarshaller to encode it as a float there. I'm a bit more inclined to convert it to float ASAP than handling it on xmlrpc layer. @mikem ?

mikem commented 2 years ago

Change EXTRACT() to return type numeric instead of float8 (Peter Eisentraut)

This avoids loss-of-precision issues in some usages. The old behavior can still be obtained by using the old underlying function date_part().

So, seems like we could change EXTRACT(EPOCH FROM $field) to date_part('epoch', $field).

That's a bit more invasive than the customized adapter, but might be more broadly compatible overall (assuming my reading is correct and that date_part really still returns floats).

I agree that we should address this when the data comes out of the db rather than wait for xmlrpc marshalling. The latter would risk breaking code inside the hub.

Edited 2 years ago by mikem