@cverna @otaylor @jkaluza

Could this be fallout from my moving odcs storage to nfs? Or something between osbs and odcs? Or ?

The error:

Image build failed. Error in plugin resolve_module_compose: HTTPError('401 Client Error: Unauthorized for url: https://odcs.fedoraproject.org/api/1/composes/'). OSBS build id: foobillard-master-e3d1b-3

makes it pretty clear that something is going wrong with how ODCS authenticates to OSBS. Can we check to see if the OIDC token that OSBS uses for this might have expired?

I am looking at that

@limb I have renewed the token can you give it a try ? and if that works can you please close the ticket

Thanks

It fails on timeout for me now:
https://koji.fedoraproject.org/koji/taskinfo?taskID=37286923

It fails on timeout for me now:
https://koji.fedoraproject.org/koji/taskinfo?taskID=37286923

Hum not sure what is going here, it looks like the authentication issue is now solved but it is timing out trying to get the results of the compose, even if https://odcs.fedoraproject.org/api/1/composes/511 seems to be there.

It fails on timeout for me now:
https://koji.fedoraproject.org/koji/taskinfo?taskID=37286923

Hum not sure what is going here, it looks like the authentication issue is now solved but it is timing out trying to get the results of the compose, even if https://odcs.fedoraproject.org/api/1/composes/511 seems to be there.

Ha the composes is there but the status of it is wait. So that maybe something stuck in ODCS.

@jkaluza could you give us an hand looking at that ^^

A few new errors for different builds:

Image build failed. Error in plugin resolve_module_compose: RuntimeError("Failed request for compose_id=512: Error while generating compose: [Errno 30] Read-only file system: '/srv/odcs/koji_tag_cache'\n"). OSBS build id: endless-sky-master-caa65-3

Image build failed. Error in plugin resolve_module_compose: RuntimeError("Failed request for compose_id=513: Error while generating compose: Command '['pungi-koji', '--config=/tmp/tmp51bstrbt/pungi.conf', '--target-dir=/srv/odcs', '--nightly']' returned non-zero value 1\n"). OSBS build id: foobillard-master-e3d1b-8

Image build failed. Error in plugin resolve_module_compose: RuntimeError("Failed request for compose_id=514: Error while generating compose: Command '['pungi-koji', '--config=/tmp/tmpqy2gtl_5/pungi.conf', '--target-dir=/srv/odcs', '--nightly']' returned non-zero value 1\n"). OSBS build id: ricochet-master-f4ee6-2

https://odcs.fedoraproject.org/api/1/composes/511 is still in state 0 - WAIT, so it's definitely a ODCS-side issue. Might be the same thing as the errors that @limb is seeing - which sounds like @kevin: "fallout from my moving odcs storage to nfs?"

NFS mounts were set up to be ro to read from /mnt/koji. This permission got inherited by the odcs mount causing it to set the drive to ro after boot. Changes to /etc/fstab were made to make this not happen and then an additional fix in the playbook.

Build was tested and successfully completed. Contacted otaylor who confirmed ticket could be closed.

Thank you!

As I mentioned failure above, I'm reporting success this time. Thanks for fixing it.