Issue #7918: Container compose keeps failing because of armhfp build - releng

releng

#7918 Container compose keeps failing because of armhfp build

Closed: Fixed 5 years ago Opened 5 years ago by cverna.

Describe the issue

We still did not release f29 container on registry.fp.o since the last valid compose is from 2018-10-28.

the armhfp build needs to be fixed since it cannot be failable see (https://pagure.io/pungi-fedora/pull-request/671)

When do you need this? (YYYY/MM/DD)
When is this no longer needed or useful? (YYYY/MM/DD)
If we cannot complete your request, what is the impact?

mohanboddu commented 5 years ago

Once @ausil gets back we should ask him to update arm01.pia.fp.o to 29:

https://pagure.io/dusty/failed-composes/issue/1024#comment-541033

mohanboddu commented 5 years ago

Now this has become a missing kernel issue, I will try and find what made it skip kernel installation.

pbrobinson commented 5 years ago

@mohanboddu the fixes are on their way in, we're waiting for a new lorax build and it should be OK. The breakage was due to changes in grubby and needing grubby-deprecated instead.

mohanboddu commented 5 years ago

@pbrobinson Oh, thanks for the info.

kevin commented 5 years ago

I spent time today trying to track this down further.

First, I updated and rebooted the armv7 container builder to make sure it had the latest everything.

Next, since the armv7 container builder was updated to f29, it had a version of oz installed that did not have our patch to use a virt rng device, so it took almost the entire time for it to boot.

So, I fixed oz (this time in rawhide also) to have the patch and updated it.

Then, we get:

Dec 10 23:33:01 localhost systemd[1]: Starting dracut initqueue hook...                                                
Dec 10 23:33:02 localhost systemd[1]: Reloading.                                                                      
Dec 10 23:33:02 localhost iscsid[807]: iSCSI daemon with pid=808 started!                                              
Dec 10 23:33:04 localhost dracut-initqueue[810]: RTNETLINK answers: File exists                                        
Dec 10 23:34:07 localhost kernel: hrtimer: interrupt took 6124167 ns                                         
Dec 10 23:36:36 localhost dracut-initqueue[810]: Warning: dracut-initqueue timeout - starting timeout scripts
...tons more of these...
Dec 10 23:38:24 localhost systemd[1]: Starting Setup Virtual Console...
Dec 10 23:38:24 localhost systemd-vconsole-setup[4145]: KD_FONT_OP_GET failed while trying to get the font metadata: Fu
nction not implemented
Dec 10 23:38:24 localhost systemd-vconsole-setup[4145]: Fonts will not be copied to remaining consoles
Dec 10 23:38:24 localhost systemd[1]: Started Setup Virtual Console.
Dec 10 23:38:24 localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=syst
emd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 10 23:38:24 localhost kernel: audit: type=1130 audit(1544485104.647:15): pid=1 uid=0 auid=4294967295 ses=4294967295
 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=
? res=success'
Dec 10 23:38:24 localhost audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=syste
md-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 10 23:38:24 localhost kernel: audit: type=1131 audit(1544485104.676:16): pid=1 uid=0 auid=4294967295 ses=4294967295
 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=
? res=success'
Dec 10 23:38:24 localhost systemd[1]: Starting Dracut Emergency Shell...
Dec 10 23:38:24 localhost systemd[1]: Received SIGRTMIN+21 from PID 779 (plymouthd).
Dec 10 23:38:24 localhost systemd[1]: Received SIGRTMIN+21 from PID 779 (plymouthd).
Dec 10 23:38:24 localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=plym
outh-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 10 23:38:24 localhost kernel: audit: type=1130 audit(1544485104.921:17): pid=1 uid=0 auid=4294967295 ses=4294967295
 subj=kernel msg='unit=plymouth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=su
ccess'
Dec 10 23:38:24 localhost audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=plymo
uth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 10 23:38:25 localhost kernel: audit: type=1131 audit(1544485104.958:18): pid=1 uid=0 auid=4294967295 ses=4294967295
 subj=kernel msg='unit=plymouth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=su
ccess'

So, it looks like a dracut issue? There was a update on 2018-12-04 that had only 'git snapshot' as it's changelog.

Open to ideas on how to track this down more...

kevin commented 5 years ago

Looks like it's waiting for /dev/root ?

[  934.332517] dracut-initqueue[705]: /lib/dracut-lib.sh@444(check_finished): local f                                  
[  934.333934] dracut-initqueue[705]: /lib/dracut-lib.sh@445(check_finished): for f in $hookdir/initqueue/finished/*.s$
[  934.360360] dracut-initqueue[705]: /lib/dracut-lib.sh@446(check_finished): '[' '/lib/dracut/hooks/initqueue/finishe$
/devexists-\x2fdev\x2froot.sh' = '/lib/dracut/hooks/initqueue/finished/*.sh' ']'                                       
[  934.362553] dracut-initqueue[705]: /lib/dracut-lib.sh@447(check_finished): '[' -e '/lib/dracut/hooks/initqueue/fini$
hed/devexists-\x2fdev\x2froot.sh' ']'                                                                                  
[  934.364894] dracut-initqueue[705]: /lib/dracut-lib.sh@447(check_finished): . '/lib/dracut/hooks/initqueue/finished/$
evexists-\x2fdev\x2froot.sh'                                                                                           
[  934.367848] dracut-initqueue[705]: //lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2froot.sh@1(source): '[$
 -e /dev/root ']'                                                                                                      
[  934.370663] dracut-initqueue[705]: /lib/dracut-lib.sh@447(check_finished): return 1                                 
[  934.374725] dracut-initqueue[705]: /bin/dracut-initqueue@26(main): udevadm settle --exit-if-exists=/lib/dracut/hook$
/initqueue/work                                                                                                        
[  934.776785] dracut-initqueue[705]: /bin/dracut-initqueue@28(main): check_finished

kevin commented 5 years ago

[18:12:34] <pwhalen> nirik, there is another bug - armhfp has an issue with virt-3.0, needs to be changed to virt-2.11
[18:15:17] <pwhalen> in /usr/lib/python2.7/site-packages/oz/Guest.py, line is "mods = {'arch': self.tdl.arch, 'machine': 'virt'}" change to virt-2.11 and it should work

Changing that and it completed finally.

@pwhalen is there a bug on that we can follow?