https://odcs.stg.fedoraproject.org/api/1/composes/230
shows:
"Error while generating compose: No active exception to reraise\n"
This blocks further progress on testing Flatpaks support in Bodhi (and any other Flatpak testing in Bodhi.)
So, some more background here:
I ran the master playbook yesterday, so likely it ran any ansible config that was in place for odcs.
I note that some packages were updated early yesterday: python3-mod_wsgi-4.5.20-4.fc28.x86_64 Thu 27 Sep 2018 09:46:27 PM UTC python3-odcs-common-0.2.7-1.fc28.noarch Thu 27 Sep 2018 09:46:19 PM UTC python3-psycopg2-2.7.4-1.fc28.x86_64 Thu 27 Sep 2018 09:46:15 PM UTC python2-libs-2.7.15-3.fc28.x86_64 Thu 27 Sep 2018 04:36:11 AM UTC python2-2.7.15-3.fc28.x86_64 Thu 27 Sep 2018 04:36:11 AM UTC lcms2-2.9-4.fc28.x86_64 Thu 27 Sep 2018 04:36:11 AM UTC
apache log on frontend has: [Fri Sep 28 16:49:46.930690 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] mod_wsgi (pid=12370): Exception occurred processing WSGI script '/usr/share/odcs/odcs.wsgi'. [Fri Sep 28 16:49:46.930862 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] Traceback (most recent call last): [Fri Sep 28 16:49:46.930997 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] File "/usr/share/odcs/odcs.wsgi", line 6, in <module> [Fri Sep 28 16:49:46.931078 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] from odcs.server import app as application [Fri Sep 28 16:49:46.931187 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] ModuleNotFoundError: No module named 'odcs.server'
Adding/assigning @jkaluza here and cc @ralph as he was working on it yesterday too.
Metadata Update from @kevin: - Issue assigned to jkaluza - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: odcs
Metadata Update from @mizdebsk: - Issue tagged with: staging
This should be fixed now, but your compose still fails, because there is no "banner" module in Koji staging anymore - it probably was removed during the last Koji stage sync.
Thanks!
Note that prod has some issues too... on the frontend:
Oct 01 15:16:24 odcs-backend01.phx2.fedoraproject.org /usr/bin/odcs-backend[29800]: Exception in backend thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/odcs/server/backend.py", line 76, in _run self.do_work() File "/usr/lib/python2.7/site-packages/odcs/server/backend.py", line 215, in do_work self._remove_compose_dir(path) File "/usr/lib/python2.7/site-packages/odcs/server/backend.py", line 148, in _remove_compose_dir shutil.rmtree(toplevel_dir) File "/usr/lib64/python2.7/shutil.py", line 261, in rmtree rmtree(fullname, ignore_errors, onerror) File "/usr/lib64/python2.7/shutil.py", line 261, in rmtree rmtree(fullname, ignore_errors, onerror) File "/usr/lib64/python2.7/shutil.py", line 270, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.7/shutil.py", line 268, in rmtree os.rmdir(path) OSError: [Errno 13] Permission denied: '/srv/odcs/odcs-68-1-20180927.n.0/work/src'
can you take a look at prod as well?
Yes, I plan to update prod tomorrow.
@kevin: I'm pospoting the update after email discussion with @otaylor. We are going to fix one more issue on staging, he will test it and then we will deploy prod.
Both prod and staging odcs instances are updated to latest versions and should be working, waiting for @otaylor to verify.
Staging is not working - https://odcs.stg.fedoraproject.org/api/1/composes/238 - either the backend isn't running or is erroring out.
Production also seems to have issues:
https://odcs.fedoraproject.org/api/1/composes/99
"Error while generating compose: '<' not supported between instances of 'NoneType' and 'NoneType'\n"
Strangely - it seems to have completed successfully first - https://kojipkgs.fedoraproject.org//work/tasks/4645/30064645/orchestrator.log shows:
Retrieved compose information for compose_id=99: {'arches': 'x86_64', 'flags': [], 'id': 99, 'koji_event': None, 'koji_task_id': None, 'multilib_arches': '', 'multilib_method': 0, 'owner': 'osbs@service', 'packages': None, 'removed_by': None, 'result_repo': 'https://odcs.fedoraproject.org/composes/latest-odcs-99-1/compose/Temporary', 'result_repofile': 'https://odcs.fedoraproject.org/composes/latest-odcs-99-1/compose/Temporary/odcs-99.repo', 'results': ['repository'], 'sigkeys': '', 'source': 'flatpak-runtime:f29:20180928134930:6c81f848', 'source_type': 2, 'state': 2, 'state_name': 'done', 'state_reason': 'Compose is generated successfully', 'time_done': '2018-10-05T15:55:12Z', 'time_removed': None, 'time_submitted': '2018-10-05T15:54:57Z', 'time_to_expire': '2018-10-06T15:54:57Z'}
From IRC just now:
17:12:50 mizdebsk │ otaylor, prod is updated too 17:12:58 mizdebsk │ and my compose just completed 17:15:20 otaylor │ mizdebsk: Great. As far as I can tell, ODCS is doing what we need it to do :-)
Metadata Update from @ralph: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I've just deployed odcs-0.2.15-2.fc28 in both staging and production, which resolved all issues.
odcs-0.2.15-2.fc28
Login to comment on this ticket.