releng

Clone

#11095 Please help me unstuck armv7hl Python builds
Closed: It's all good 11 months ago by churchyard. Opened 2 years ago by churchyard.

Closed: It's all good
Reopen Issue
  • Describe the issue

I have 3 Koji builds running:

They all keep restarting the armv7hl task, some of them for 135 hours now.
I need the builds to finish, as they contain some security bugfixes.

Other Python builds succeeded, on the following builders:

  • buildvm-a32-05.iad2.fedoraproject.org
  • buildvm-a32-29.iad2.fedoraproject.org
  • buildvm-a32-33.iad2.fedoraproject.org

Could you please migrate the armv7hl task of the linked builds to the listed builders to see if it helps?

  • When do you need this? sooner the better

  • When is this no longer needed or useful? when the builds finish

  • If we cannot complete your request, what is the impact? cannot update Pythons on Fedora 35/36.

Thanks.

it seems to be building now, what I did is:

$ koji free-task 92962486 93135891 92962400

That will pick the next free builder and try to build there. I will keep an eye on the builds and pin them to one of the builders if they get stuck again.

Metadata Update from @humaton:
- Issue tagged with: high-gain, medium-trouble, ops
2 years ago

it seems to be building now,

It's always building. I have tried running free-task as well, it did not help.

Metadata Update from @churchyard:
- Issue untagged with: high-gain, medium-trouble, ops
2 years ago

Metadata Update from @phsmoura:
- Issue tagged with: high-gain, medium-trouble, ops
2 years ago

One of them apparently finished yesterday. Two remaining are currently building on mentioned hosts.

Hmm, they keep restarting on any host.

The one succesfull build finished on buildvm-a32-26.iad2.fedoraproject.org.

Could you please move the two remaining builds to one of the listed "good" builders to see if it helps?

I tried that and didn't work. let's see about buildvm-a32-26.iad2.fedoraproject.org.

I tried attaching the tasks to different workers, but it didn't help tasks restart and randomly finish.

One of the 2 remaining builds failed with BuildrootError: Requested repo (4852834) is DELETED, restarted as https://koji.fedoraproject.org/koji/taskinfo?taskID=93243486

Eventually the builds usually fail with koji repos 404 or similar errors, so I keep submitting new ones, but they seem to keep restarting. I've been unable to build Python on 32bit ARM for a ~week and I don't know what I should do.

In the worst case scenario, we can disable the tests on 32bit ARM and stop caring whether it works there or not. Considering the importance of Python I am not very keen on doing it, at least for Python 3.10 on F35/36.

Anyway, current builds:

https://koji.fedoraproject.org/koji/taskinfo?taskID=93411465
https://koji.fedoraproject.org/koji/taskinfo?taskID=93411466
https://koji.fedoraproject.org/koji/taskinfo?taskID=93423703
https://koji.fedoraproject.org/koji/taskinfo?taskID=93423704

Sorry for arriving here late. I was on PTO last week and am fighting off covid this week. ;)

Anyhow, the problem is finding a kernel that doesn't oom kill things quickly.

They are all currently running 5.19.8-200.fc36.armv7hl+lpae which has proved great with other builds, but seems to kill python builds. ;(

Are they all dying in tests? Is there any way to reduce the resources for tests? less threads? or less memory ideally?

There's a number of things we could try:

  • newer kernel? say 5.19.17 or even 6.0.x ?
  • drop memory down to 4gb and use a non lpae kernel. This is likely to be super slow, but might be more reliable?

Are they all dying in tests?

I have no idea. The way it behaves prevents me to see where exactly it gets restarted. When observing the build log during build, it's usually in %build but I have seen the tests run in them right now in https://koji.fedoraproject.org/koji/taskinfo?taskID=93411520 (which runs for 50 minutes now, quite unusual).

Is there any way to reduce the resources for tests? less threads? or less memory ideally?

I've tried changing the test jobs from -j0 (all CPUs) to -j%{_smp_build_ncpus} and setting %_smp_build_ncpus to 3 or even 1, but it keeps restartign anyway.

I could try anything at this point. 4 GB of memory should be enough to build Python.

I made a snafu when trying to set %_smp_build_ncpus to 3, so trying again, but given the -j1 build was done the right way and keeps getting restarted, I don't expect -j3 to be any better anyway.

When observing the build log during build, it's usually in %build but I have seen the tests run in them right now in https://koji.fedoraproject.org/koji/taskinfo?taskID=93411520 (which runs for 50 minutes now, quite unusual).

OK, it now clearly restarted during %check.

ok. I upgraded buildvm-a32-10 to use 6.0.5-200.fc36.armv7hl+lpae and force assigned the python 3.11 build there. Lets see how that goes.

ok, That hit oom. Last bits in the build.log: 

...
0:11:58 load avg: 11.38 [297/436] test_sax passed -- running: test_pickle (2 min 10 sec), test_multiprocessing_spawn (3 min 21 sec), test_gdb (6 min 27 sec)                                               
0:11:58 load avg: 11.11 [298/436] test_scope passed -- running: test_pickle (2 min 11 sec), test_multiprocessing_spawn (3 min 22 sec), test_gdb (6 min 28 sec)                                             
0:12:00 load avg: 11.11 [299/436] test_secrets passed -- running: test_pickle (2 min 13 sec), test_multiprocessing_spawn (3 min 24 sec), test_gdb (6 min 29 sec)                                           
0:12:01 load avg: 11.11 [300/436] test_script_helper passed -- running: test_pickle (2 min 14 sec), test_multiprocessing_spawn (3 min 25 sec), test_gdb (6 min 31 sec)                                     
0:12:04 load avg: 11.02 [301/436] test_select passed -- running: test_pickle (2 min 16 sec), test_multiprocessing_spawn (3 min 27 sec), test_gdb (6 min 33 sec)                                            
0:12:15 load avg: 10.87 [302/436] test_gdb passed (6 min 43 sec) -- running: test_pickle (2 min 28 sec), test_regrtest (30.4 sec), test_multiprocessing_spawn (3 min 39 sec)                               
0:12:17 load avg: 10.87 [303/436] test_setcomps passed -- running: test_pickle (2 min 30 sec), test_regrtest (32.4 sec), test_multiprocessing_spawn (3 min 41 sec)

Now installing 6.0.5-200.fc36.armv7hl on 10 and will see if 4gb is enough to work.

So, that worked, but... it's not a great solution because that setup causes a ton of larger other packages to start failing (libreoffice, ceph, gcc, etc).

But I guess at least we have a workaround now. I'll feed your other builds to 10.

Awesome, thanks.

it's not a great solution because that setup causes a ton of larger other packages to start failing (libreoffice, ceph, gcc, etc).

Can a builder have a denylist, or something like that?

In the future, how do I ensure I get this one particular builder?

Sorry for the delay here.

So, koji doesn't have anything like a denylist... the way it works is that koji has 'channels' and specific builders can subscribe to specific channels. Then, the hub has a policy config that can say specific build names or whatever go to a specific channel. So, we could do this by setting a few of the a32 builders up like this, subscribing them to a new channel and then specifically directing just python builds to those builders. I was really hoping to avoid doing that however. It means mantaining policy and making some a32 builders useless except for when python builds are happening, etc.

I'm not sure fully what else to try to avoid it however. :(

Some questions:

How often do you do these builds (f35/f36?

Do you have a specific/exact list of packages affected? We can use globs, but obviously 'python*' will not just catch these. ;)

How many builds do you usually do at once? (ie, how many builders do we need to allocate to this? I suppose even just 1 would be better than never completing).

Can this wait for after f37 release/freeze is over?

How often do you do these builds (f35/f36?

Let me ignore f35 for now. It's going to be EOL soon and if I count it, we would need to half the results in 5 weeks anyway. Let's assume just assume the numbers will be more or less double in size for the next month.

Since 2022-05-08 i.e. for half a year, there were:

  • 12 builds of python3.11
  • 6 builds of python3.10
  • 4 builds of python3.9
  • 3 builds of python3.8
  • 3 builds of python3.7
  • 3 builds of python3.6
  • 1 build of python2.7
  • 2 builds of 3 pypys = 6 builds

I am not entirely sure if python/pypy 2 is affected, but pypy3 is. Python 2 makes little difference.

We must also consider that the most updated package was python3.11 which is now released and the most frequently built thing will be python3.12 which is excluding arm32. So I will count python3.11 as 6 instead, which is more likely for the future half a year.

That is 6+6+4+3+3+3+1+6=32 updates for 6 months. Each update is built on Zuul, on Fedora CI, and when it is actually shipped. That is 3 builds for one update assuming the packager does not rebase the PR. In reality, we often do. Sometimes, we submit our own scratchbuilds, but rarely on non-rawhide. Let's be optimistic and count 32*3=96 build per 6 months, that is on average 16 builds a month on f36.

Do you have a specific/exact list of packages affected?

See above. pypys are: pypy, pypy3.6, pypy3.7 and pypy3.9

Good globs might be: python?.* pypy*

How many builds do you usually do at once?

Usually 1, 2, or 3 on f36. However, happy to do them sequentially.

The security releases (3.6, 3.7, 3.8) often have releases on 1 date. PyPy always have releases on the same date.

Can this wait for after f37 release/freeze is over?

As long as you can help me ad hoc when we will be shipping the next CVE fixes, yes. There are several new CVE bugzillas opened for at least 2 problems affecting almost all of the versions.
Edited a year ago by churchyard

(We might not have the fixes ready before F37 release anyway.)

ok. So, perhaps I setup 4 builders like this then... that might sometimes mean some wait until another finishes, but mostly should handle it.

Yes, happy to help if there's builds before we get this offically setup.

Assigned those 3 to buildvm-a32-10.

Thanks.

One succeeded on buildvm-a32-07 and I free-tasked the other one until it got buildvm-a32-10.

Builder 10 seems likely to have an issue. ;(

See https://bodhi.fedoraproject.org/updates/FEDORA-2022-f44dd1bec2

It seems the armv7 build somehow completed multiple times and the later one overwrote the eariler one in the db and so sizes are mismatched. ;(

I'm not sure how this is possible tho, as the first time it finished the build should be in the db and it should reject later ones as the same n-v-r.

Did you have multiple builds for that same package running at the same time?

CC: @mikem @tkopecek

If you look at: 'koji --debug download-build --arch=noarch python3.10-3.10.8-2.fc35' it gives:

Downloaded rpm python-unversioned-command-3.10.8-2.fc35.noarch.rpm size 10208 does not match db size 10206, deleting

I can see in logs on that arm builder:

Nov 09 21:30:23 buildvm-a32-10.iad2.fedoraproject.org kojid[1121]: 2022-11-09 21:30:23,573 [INFO] {1121} koji.TaskManager:1471 RESPONSE: "<?xml version='1.0'?>\n<methodResponse>\n<params>\n<param>\n<value><struct>\n<member>\n<name>rpms</name>\n<value><array><data>\n<value><string>tasks/2416/93982416/python3.10-debugsource-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3.10-debuginfo-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-libs-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-test-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-debug-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-idle-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-tkinter-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-devel-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python-unversioned-command-3.10.8-2.fc35.noarch.rpm</string></value>\n</data></array></value>\n</member>\n<member>\n<name>srpms</name>\n<value><array><data>\n<value><string>tasks/2416/93982416/python3.10-3.10.8-2.fc35.src.rpm</string></value>\n</data></array></value>\n</member>\n<member>\n<name>logs</name>\n<value><array><data>\n<value><string>tasks/2416/93982416/hw_info.log</string></value>\n<value><string>tasks/2416/93982416/state.log</string></value>\n<value><string>tasks/2416/93982416/build.log</string></value>\n<value><string>tasks/2416/93982416/root.log</string></value>\n<value><string>tasks/2416/93982416/mock_output.log</string></value>\n<value><string>tasks/2416/93982416/noarch_rpmdiff.json</string></value>\n</data></array></value>\n</member>\n<member>\n<name>brootid</name>\n<value><int>39173869</int></value>\n</member>\n</struct></value>\n</param>\n</params>\n</methodResponse>\n"
Nov 09 23:08:16 buildvm-a32-10.iad2.fedoraproject.org kojid[18246]: 2022-11-09 23:08:16,417 [INFO] {18246} koji.TaskManager:1471 RESPONSE: "<?xml version='1.0'?>\n<methodResponse>\n<params>\n<param>\n<value><struct>\n<member>\n<name>rpms</name>\n<value><array><data>\n<value><string>tasks/2416/93982416/python3.10-debugsource-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3.10-debuginfo-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-libs-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-test-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-debug-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-idle-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-tkinter-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-devel-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python3-3.10.8-2.fc35.armv7hl.rpm</string></value>\n<value><string>tasks/2416/93982416/python-unversioned-command-3.10.8-2.fc35.noarch.rpm</string></value>\n</data></array></value>\n</member>\n<member>\n<name>srpms</name>\n<value><array><data>\n<value><string>tasks/2416/93982416/python3.10-3.10.8-2.fc35.src.rpm</string></value>\n</data></array></value>\n</member>\n<member>\n<name>logs</name>\n<value><array><data>\n<value><string>tasks/2416/93982416/hw_info.log</string></value>\n<value><string>tasks/2416/93982416/state.log</string></value>\n<value><string>tasks/2416/93982416/build.log</string></value>\n<value><string>tasks/2416/93982416/root.log</string></value>\n<value><string>tasks/2416/93982416/mock_output.log</string></value>\n<value><string>tasks/2416/93982416/noarch_rpmdiff.json</string></value>\n</data></array></value>\n</member>\n<member>\n<name>brootid</name>\n<value><int>39174844</int></value>\n</member>\n</struct></value>\n</param>\n</params>\n</methodResponse>\n"

Any ideas whats going on here?

First python-unversioned-command (in db) has buildtime (2022, 11, 9, 21, 29, 31), while second (downloaded from koji) has buildtime 23:08:29, so evidently different things.

We've hit some FUN (in dwarfortress sense). It seems that first task "won" db record and finished successfully - then it was killed and continued in the (already finished) task after restart.

See, that parent task https://koji.fedoraproject.org/koji/taskinfo?taskID=93982395 finished at 21:30, while child task https://koji.fedoraproject.org/koji/taskinfo?taskID=93982416 (re)started last time at 22:04 (it has restarted three times, finally ending at 23:08). If the child was allowed to restart after parent is finished it will upload the results overriding work directory (which is still valid in that point) thus rewriting also the final output. Definitely a koji bug but I'm now not sure how we will fix this (probably need some fix in in testing actual state of task)

Created https://pagure.io/koji/issue/3583
Edited a year ago by tkopecek

Ugh. I thought koji would clear the work dir anytime something was restarted/canceled/etc. :(

So, what do we do here? Is there any way of fixing the build in the db? or should we just abandon it and bump and rebuild?

Do you want/need/have a koji ticket to track this issue?

You can fix it in the db - but it means updating (probably) all rpm data for given build. Rebuild is definitely safer. There is https://pagure.io/koji/issue/3583 for koji.

A rebuild will most likely result in yet another restarting spree.

There is a regression in 3.10.8 and I'm gonna patch and build it again anyway: https://bugzilla.redhat.com/show_bug.cgi?id=2142602

I've put in a freeze break to reboot all the a32 builders with the 6.0.x kernel. I'm hoping that it will handle things better than the current 5.x one.

If you could hold off a bit until I get that done we can try these builds on the 6.0.x kernel.

Builds succeeded on builders 07 and 09.

Rebuild is definitely safer.

A rebuild landed in bodhi, all seems good wrt that.

ok, glad they all worked... not sure why. :)

In any case I have rebooted them all into 6.0.x. If you could perhaps try some scratch builds of ones that gave you problems before and see if the new kernel is handling it any better?

Or should we just wait until you have need to do another round of updates?

I've started the original builds listed in the initial comment of this issue as scratch builds:

I've started the original builds listed in the initial comment of this issue as scratch builds:

no restart, buildvm-a32-27.iad2.fedoraproject.org

no restart, buildvm-a32-09.iad2.fedoraproject.org

2 restarts so far on buildvm-a32-14.iad2.fedoraproject.org and buildvm-a32-23.iad2.fedoraproject.org -- currently at buildvm-a32-09.iad2.fedoraproject.org

2 restarts so far on buildvm-a32-14.iad2.fedoraproject.org and buildvm-a32-23.iad2.fedoraproject.org -- currently at buildvm-a32-09.iad2.fedoraproject.org

Finished at buildvm-a32-09.iad2.fedoraproject.org

A bit off-topic - with scratch builds we can confirm just that restart happens (which is already some info) but not that first version is overriden by the later as there is no db record for scratch build.
Edited a year ago by tkopecek

@kevin In https://bodhi.fedoraproject.org/updates/FEDORA-2022-b17bf30e88 you've asked me to rebuild it once more. The build however is once again stuck in a restart loop https://koji.fedoraproject.org/koji/taskinfo?taskID=94265140

@kevin In https://bodhi.fedoraproject.org/updates/FEDORA-2022-b17bf30e88 you've asked me to rebuild it once more. The build however is once again stuck in a restart loop https://koji.fedoraproject.org/koji/taskinfo?taskID=94265140

It built.

Ugh. I was hoping it was fixed.

ok, so I rebooted buildvm-a32-30 and 31 into non lpae kernels.
I then assigned those two jobs to them.

Hopefully that will finish.

I guess I can look at setting up a seperate channel for these, but seems like a lot of pain for something that will be going away soon.
I'll see if I can come up with a more clever way to do it...

The fc35 build seems to actually hang in test_socket. Is it possible to restart it on the same builder?

It seems to have finished now?

Indeed, thank you!

Could you please move this build https://koji.fedoraproject.org/koji/taskinfo?taskID=96049187 to whatever builder might work?

% koji assign-task -f 96049187 buildvm-a32-31.iad2.fedoraproject.org
assigned task 96049187 to host buildvm-a32-31.iad2.fedoraproject.org

Thanks. Next time I'll check if I am able to do that myself.

https://koji.fedoraproject.org/koji/taskinfo?taskID=97391987
https://koji.fedoraproject.org/koji/taskinfo?taskID=97391899

Those two are running for 35+ hours. The entire build is blocked by s390x anyway, so no ruish, but they need to eventually finish.

I've run koji assign-task -f 97391987 buildvm-a32-31.iad2.fedoraproject.org and it moved the build, but it still restarted apparently. Will try the other one now. EDIT: no dice.
Edited a year ago by churchyard

ok. I made sure 33 and 32 were running the non lpae kernel and force assigned the two jobs to them.

Lets see if they complete now.

BuildrootError: could not init mock buildroot, mock exited with status 30; see root.log for more information

Error: Transaction test error:
  package filesystem-3.18-2.fc36.armv7hl is intended for a different architecture
  package coreutils-common-9.0-9.fc36.armv7hl is intended for a different architecture
  package libgcc-12.2.1-4.fc36.armv7hl is intended for a different architecture
  package glibc-minimal-langpack-2.35-22.fc36.armv7hl is intended for a different architecture
  package glibc-common-2.35-22.fc36.armv7hl is intended for a different architecture
  package glibc-2.35-22.fc36.armv7hl is intended for a different architecture
  package ncurses-libs-6.2-9.20210508.fc36.armv7hl is intended for a different architecture
  package bash-5.2.15-1.fc36.armv7hl is intended for a different architecture
  package zlib-1.2.11-33.fc36.armv7hl is intended for a different architecture
  package bzip2-libs-1.0.8-11.fc36.armv7hl is intended for a different architecture
  package xz-libs-5.4.1-1.fc36.armv7hl is intended for a different architecture
  package libzstd-1.5.2-2.fc36.armv7hl is intended for a different architecture
  package sqlite-libs-3.36.0-5.fc36.armv7hl is intended for a different architecture
  package libcap-2.48-4.fc36.armv7hl is intended for a different architecture
  package gmp-1:6.2.1-2.fc36.armv7hl is intended for a different architecture
  package libgpg-error-1.45-1.fc36.armv7hl is intended for a different architecture
  package popt-1.18-7.fc36.armv7hl is intended for a different architecture
  package libxml2-2.10.3-2.fc36.armv7hl is intended for a different architecture
  package libstdc++-12.2.1-4.fc36.armv7hl is intended for a different architecture
  package lua-libs-5.4.4-7.fc36.armv7hl is intended for a different architecture
  package elfutils-libelf-0.188-3.fc36.armv7hl is intended for a different architecture
  package file-libs-5.41-4.fc36.armv7hl is intended for a different architecture
  package readline-8.2-2.fc36.armv7hl is intended for a different architecture
  package libattr-2.5.1-4.fc36.armv7hl is intended for a different architecture
  package libacl-2.3.1-3.fc36.armv7hl is intended for a different architecture
  package libcom_err-1.46.5-2.fc36.armv7hl is intended for a different architecture
  package libffi-3.4.2-8.fc36.armv7hl is intended for a different architecture
  package p11-kit-0.24.1-2.fc36.armv7hl is intended for a different architecture
  package libunistring-1.0-1.fc36.armv7hl is intended for a different architecture
  package libidn2-2.3.4-1.fc36.armv7hl is intended for a different architecture
  package libuuid-2.38-1.fc36.armv7hl is intended for a different architecture
  package libxcrypt-4.4.33-4.fc36.armv7hl is intended for a different architecture
  package libassuan-2.5.5-4.fc36.armv7hl is intended for a different architecture
  package libgcrypt-1.10.1-3.fc36.armv7hl is intended for a different architecture
  package expat-2.5.0-1.fc36.armv7hl is intended for a different architecture
  package gdbm-libs-1:1.22-2.fc36.armv7hl is intended for a different architecture
  package json-c-0.15-3.fc36.armv7hl is intended for a different architecture
  package keyutils-libs-1.6.1-4.fc36.armv7hl is intended for a different architecture
  package libsepol-3.3-3.fc36.armv7hl is intended for a different architecture
  package libsmartcols-2.38-1.fc36.armv7hl is intended for a different architecture
  package libtasn1-4.19.0-1.fc36.armv7hl is intended for a different architecture
  package lz4-libs-1.9.3-4.fc36.armv7hl is intended for a different architecture
  package pcre-8.45-1.fc36.1.armv7hl is intended for a different architecture
  package elfutils-libs-0.188-3.fc36.armv7hl is intended for a different architecture
  package grep-3.7-2.fc36.armv7hl is intended for a different architecture
  package systemd-libs-250.9-1.fc36.armv7hl is intended for a different architecture
  package dbus-libs-1:1.14.4-1.fc36.armv7hl is intended for a different architecture
  package libcomps-0.1.18-2.fc36.armv7hl is intended for a different architecture
  package libpsl-0.21.1-5.fc36.armv7hl is intended for a different architecture
  package mpdecimal-2.5.1-3.fc36.armv7hl is intended for a different architecture
  package libksba-1.6.3-1.fc36.armv7hl is intended for a different architecture
  package mpfr-4.1.0-9.fc36.armv7hl is intended for a different architecture
  package nettle-3.8-1.fc36.armv7hl is intended for a different architecture
  package alternatives-1.21-1.fc36.armv7hl is intended for a different architecture
  package p11-kit-trust-0.24.1-2.fc36.armv7hl is intended for a different architecture
  package gnutls-3.7.8-3.fc36.armv7hl is intended for a different architecture
  package libbrotli-1.0.9-7.fc36.armv7hl is intended for a different architecture
  package libcap-ng-0.8.3-1.fc36.armv7hl is intended for a different architecture
  package audit-libs-3.0.9-1.fc36.armv7hl is intended for a different architecture
  package libgomp-12.2.1-4.fc36.armv7hl is intended for a different architecture
  package libnghttp2-1.51.0-1.fc36.armv7hl is intended for a different architecture
  package libsigsegv-2.14-2.fc36.armv7hl is intended for a different architecture
  package gawk-5.1.1-2.fc36.armv7hl is intended for a different architecture
  package libverto-0.3.2-3.fc36.armv7hl is intended for a different architecture
  package libyaml-0.2.5-7.fc36.armv7hl is intended for a different architecture
  package npth-1.6-8.fc36.armv7hl is intended for a different architecture
  package pcre2-10.40-1.fc36.armv7hl is intended for a different architecture
  package libselinux-3.3-4.fc36.armv7hl is intended for a different architecture
  package sed-4.8-10.fc36.armv7hl is intended for a different architecture
  package openssl-libs-1:3.0.5-2.fc36.armv7hl is intended for a different architecture
  package coreutils-9.0-9.fc36.armv7hl is intended for a different architecture
  package krb5-libs-1.19.2-12.fc36.armv7hl is intended for a different architecture
  package libtirpc-1.3.3-0.fc36.armv7hl is intended for a different architecture
  package libfsverity-1.4-7.fc36.armv7hl is intended for a different architecture
  package zchunk-libs-1.2.3-1.fc36.armv7hl is intended for a different architecture
  package libnsl2-2.0.0-3.fc36.armv7hl is intended for a different architecture
  package python3-3.10.9-1.fc36.armv7hl is intended for a different architecture
  package python3-libs-3.10.9-1.fc36.armv7hl is intended for a different architecture
  package python3-libcomps-0.1.18-2.fc36.armv7hl is intended for a different architecture
  package cyrus-sasl-lib-2.1.27-18.fc36.armv7hl is intended for a different architecture
  package libssh-0.9.6-4.fc36.armv7hl is intended for a different architecture
  package libblkid-2.38-1.fc36.armv7hl is intended for a different architecture
  package libmount-2.38-1.fc36.armv7hl is intended for a different architecture
  package glib2-2.72.3-1.fc36.armv7hl is intended for a different architecture
  package python3-dbus-1.2.18-3.fc36.armv7hl is intended for a different architecture
  package libarchive-3.5.3-3.fc36.armv7hl is intended for a different architecture
  package libevent-2.1.12-6.fc36.armv7hl is intended for a different architecture
  package openldap-2.6.3-1.fc36.armv7hl is intended for a different architecture
  package libcurl-7.82.0-12.fc36.armv7hl is intended for a different architecture
  package gnupg2-2.3.7-3.fc36.armv7hl is intended for a different architecture
  package gpgme-1.17.0-4.fc36.armv7hl is intended for a different architecture
  package librepo-1.15.1-1.fc36.armv7hl is intended for a different architecture
  package python3-gpg-1.17.0-4.fc36.armv7hl is intended for a different architecture
  package curl-7.82.0-12.fc36.armv7hl is intended for a different architecture
  package findutils-1:4.9.0-1.fc36.armv7hl is intended for a different architecture
  package rpm-4.17.1-3.fc36.armv7hl is intended for a different architecture
  package rpm-libs-4.17.1-3.fc36.armv7hl is intended for a different architecture
  package libmodulemd-2.14.0-2.fc36.armv7hl is intended for a different architecture
  package libsolv-0.7.22-1.fc36.armv7hl is intended for a different architecture
  package libdnf-0.68.0-1.fc36.armv7hl is intended for a different architecture
  package python3-libdnf-0.68.0-1.fc36.armv7hl is intended for a different architecture
  package python3-hawkey-0.68.0-1.fc36.armv7hl is intended for a different architecture
  package rpm-build-libs-4.17.1-3.fc36.armv7hl is intended for a different architecture
  package libsemanage-3.3-3.fc36.armv7hl is intended for a different architecture
  package shadow-utils-2:4.11.1-5.fc36.armv7hl is intended for a different architecture
  package tpm2-tss-3.2.1-1.fc36.armv7hl is intended for a different architecture
  package ima-evm-utils-1.4-5.fc36.armv7hl is intended for a different architecture
  package rpm-sign-libs-4.17.1-3.fc36.armv7hl is intended for a different architecture
  package python3-rpm-4.17.1-3.fc36.armv7hl is intended for a different architecture

Seems like you assigned the tasks to aarch64 builders, judging from the hostnames.

Starting again.

Whoops. Thats what I get for trying to do too many things at once. Sorry about that. ;(

ok, lets try again. Forced to the right 32bit builders now.

Thanks.

The two builds finished successfully.

Dobby is free

Metadata Update from @churchyard:
- Issue close_status updated to: It's all good
- Issue status updated to: Closed (was: Open)
11 months ago

Login to comment on this ticket.

Metadata
None

high-gain ops medium-trouble

None
None
None
None
None
None
Boards 1
Ops Status: Backlog
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.