fedora-infrastructure

#7007 Increasing oz install timeout from 1200 seconds to 7200 seconds
Closed: Fixed Opened by sinnykumari.

Closed: Fixed
Reopen Issue
  • Describe what you need us to do:
    Cloud and container images are failing regularly on some architectures like s390x and ppc64(le) in Fedora 28 and rawhide.
    Few latest failed builds are:
    F28 cloud base - https://koji.fedoraproject.org/koji/taskinfo?taskID=27317521
    F28 container base - https://koji.fedoraproject.org/koji/taskinfo?taskID=27316756
    Rawhide cloud base - https://koji.fedoraproject.org/koji/taskinfo?taskID=27277465

By looking at oz log e.g. ppc64le cloud-base from rawhide, it looks like imagefactory install timeout is now set to 1200 seconds which causes image install stops in between due to timeout.

I recall that this was set default to 7200 seconds when koji builders were on F27.
I think install timeout change to 1200 seconds is combined effect of having configurable oz timeout patch in koji-1.15.1-1.fc28 and builders being updated to F28. We may need to set oz_install_timeout=7200 in our ansible kojid.conf file similar to we did for one of s390x builder and run the playbook on builders where image builds happen.

  • When do you need this? (YYYY/MM/DD)
    As early as possible

  • When is this no longer needed or useful? (YYYY/MM/DD)
    Always useful

  • If we cannot complete your request, what is the impact?
    container and cloud images build will keep failing in F28 and rawhide compose on some architectures like ppc64(le), s390x

thank you sinny for looking into this!

@kevin what do you think about this issue?

So, we have:

https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/kojid.conf#n28

oz_install_timeout=14400

for only s390x and aarch64. So, I guess the question is: why is s390x failing with the extra timeout?
Or is it somehow not paying attention to it? Ah, it has no boot.img in the rawhide compose at least so thats why that fails. And that in turn is caused by the sshfs mount having issues.

ok. I will just unconditionally apply this timeout on all machines and fix the s390x sshfs mount and hopefully tomorrow we will have some luck. If not, please re-open.

Pushed to ansible, playbook run and sshfs mount remounted.

:tophat:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Thanks @kevin

So, we have:
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/kojid.conf#n28
oz_install_timeout=14400
for only s390x and aarch64. So, I guess the question is: why is s390x failing with the extra timeout?
Or is it somehow not paying attention to it? Ah, it has no boot.img in the rawhide compose at least so thats why that fails. And that in turn is caused by the sshfs mount having issues.
ok. I will just unconditionally apply this timeout on all machines and fix the s390x sshfs mount and hopefully tomorrow we will have some luck. If not, please re-open.
Pushed to ansible, playbook run and sshfs mount remounted.
🎩

@kevin With this change, we had FINISHED compose for F28 container - https://kojipkgs.fedoraproject.org/compose/container/Fedora-Container-28-20180601.0/ . F28 Cloud compose from 20180601 still seems to be failing on ppc64. From ppc64 oz log, it appears that 1200 seconds install timeout is getting used on buildvm-ppc64-01.ppc.fedoraproject.org builder.

Metadata Update from @sinnykumari:
- Issue status updated to: Open (was: Closed)

I did push this, but didn't restart kojid on the compose builders. ;(

I have done so now and will refire rawhide to confirm...but hopefully it's fixed now.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Thanks @kevin ! Most of the cloud and container build looks good now after the required changes.

By looking at F28 cloud-base run from 20180603.0, It looks like ppc64 image builder buildvm-ppc64-01.ppc.fedoraproject.org is still using install timeout as 1200 seconds. Does kojid restart happened successfully on this builder?

Metadata Update from @sinnykumari:
- Issue status updated to: Open (was: Closed)

From 20180604.0 F28 cloud-base compose ppc64 log, looks like buildvm-ppc64-01.ppc.fedoraproject.org builder has now updated kojid config and image build succeeded.

Metadata Update from @sinnykumari:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.