#7270 staging ODCS: not composing
Closed: Fixed a year ago Opened a year ago by otaylor.

https://odcs.stg.fedoraproject.org/api/1/composes/230

shows:

"Error while generating compose: No active exception to reraise\n"

This blocks further progress on testing Flatpaks support in Bodhi (and any other Flatpak testing in Bodhi.)


So, some more background here:

  • I ran the master playbook yesterday, so likely it ran any ansible config that was in place for odcs.

  • I note that some packages were updated early yesterday:
    python3-mod_wsgi-4.5.20-4.fc28.x86_64 Thu 27 Sep 2018 09:46:27 PM UTC
    python3-odcs-common-0.2.7-1.fc28.noarch Thu 27 Sep 2018 09:46:19 PM UTC
    python3-psycopg2-2.7.4-1.fc28.x86_64 Thu 27 Sep 2018 09:46:15 PM UTC
    python2-libs-2.7.15-3.fc28.x86_64 Thu 27 Sep 2018 04:36:11 AM UTC
    python2-2.7.15-3.fc28.x86_64 Thu 27 Sep 2018 04:36:11 AM UTC
    lcms2-2.9-4.fc28.x86_64 Thu 27 Sep 2018 04:36:11 AM UTC

apache log on frontend has:
[Fri Sep 28 16:49:46.930690 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] mod_wsgi (pid=12370): Exception occurred processing WSGI script '/usr/share/odcs/odcs.wsgi'.
[Fri Sep 28 16:49:46.930862 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] Traceback (most recent call last):
[Fri Sep 28 16:49:46.930997 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] File "/usr/share/odcs/odcs.wsgi", line 6, in <module>
[Fri Sep 28 16:49:46.931078 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] from odcs.server import app as application
[Fri Sep 28 16:49:46.931187 2018] [wsgi:error] [pid 12370:tid 140174228756224] [remote 10.5.126.51:52646] ModuleNotFoundError: No module named 'odcs.server'

Adding/assigning @jkaluza here and cc @ralph as he was working on it yesterday too.

Metadata Update from @kevin:
- Issue assigned to jkaluza
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: odcs

a year ago

Metadata Update from @mizdebsk:
- Issue tagged with: staging

a year ago

This should be fixed now, but your compose still fails, because there is no "banner" module in Koji staging anymore - it probably was removed during the last Koji stage sync.

Thanks!

Note that prod has some issues too... on the frontend:

Oct 01 15:16:24 odcs-backend01.phx2.fedoraproject.org /usr/bin/odcs-backend[29800]: Exception in backend thread
                                                                                    Traceback (most recent call last):
                                                                                      File "/usr/lib/python2.7/site-packages/odcs/server/backend.py", line 76, in _run
                                                                                        self.do_work()
                                                                                      File "/usr/lib/python2.7/site-packages/odcs/server/backend.py", line 215, in do_work
                                                                                        self._remove_compose_dir(path)
                                                                                      File "/usr/lib/python2.7/site-packages/odcs/server/backend.py", line 148, in _remove_compose_dir
                                                                                        shutil.rmtree(toplevel_dir)
                                                                                      File "/usr/lib64/python2.7/shutil.py", line 261, in rmtree
                                                                                        rmtree(fullname, ignore_errors, onerror)
                                                                                      File "/usr/lib64/python2.7/shutil.py", line 261, in rmtree
                                                                                        rmtree(fullname, ignore_errors, onerror)
                                                                                      File "/usr/lib64/python2.7/shutil.py", line 270, in rmtree
                                                                                        onerror(os.rmdir, path, sys.exc_info())
                                                                                      File "/usr/lib64/python2.7/shutil.py", line 268, in rmtree
                                                                                        os.rmdir(path)
                                                                                    OSError: [Errno 13] Permission denied: '/srv/odcs/odcs-68-1-20180927.n.0/work/src'

can you take a look at prod as well?

Yes, I plan to update prod tomorrow.

@kevin: I'm pospoting the update after email discussion with @otaylor. We are going to fix one more issue on staging, he will test it and then we will deploy prod.

Both prod and staging odcs instances are updated to latest versions and should be working, waiting for @otaylor to verify.

Staging is not working - https://odcs.stg.fedoraproject.org/api/1/composes/238 - either the backend isn't running or is erroring out.

Production also seems to have issues:

https://odcs.fedoraproject.org/api/1/composes/99

"Error while generating compose: '<' not supported between instances of 'NoneType' and 'NoneType'\n"

Strangely - it seems to have completed successfully first - https://kojipkgs.fedoraproject.org//work/tasks/4645/30064645/orchestrator.log shows:

  Retrieved compose information for compose_id=99: 
{'arches': 'x86_64', 'flags': [], 'id': 99, 'koji_event': None, 'koji_task_id': None,
'multilib_arches': '', 'multilib_method': 0, 'owner': 'osbs@service', 'packages': None, 
'removed_by': None,
'result_repo': 'https://odcs.fedoraproject.org/composes/latest-odcs-99-1/compose/Temporary',
 'result_repofile': 'https://odcs.fedoraproject.org/composes/latest-odcs-99-1/compose/Temporary/odcs-99.repo',
'results': ['repository'], 'sigkeys': '', 'source': 'flatpak-runtime:f29:20180928134930:6c81f848', 
'source_type': 2, 'state': 2, 'state_name': 'done',
'state_reason': 'Compose is generated successfully',
'time_done': '2018-10-05T15:55:12Z', 'time_removed': None,
'time_submitted': '2018-10-05T15:54:57Z',
'time_to_expire': '2018-10-06T15:54:57Z'}

From IRC just now:

17:12:50      mizdebsk │ otaylor, prod is updated too                                                  
17:12:58      mizdebsk │ and my compose just completed                                                 
17:15:20       otaylor │ mizdebsk: Great. As far as I can tell, ODCS is doing what we need it to do :-)

Metadata Update from @ralph:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

I've just deployed odcs-0.2.15-2.fc28 in both staging and production, which resolved all issues.

Login to comment on this ticket.

Metadata