#6765 Fixing rawhide compose failure for variant Atomic on enabled arches
Closed: Fixed 6 years ago Opened 6 years ago by sinnykumari.

Rawhide nightly composes are failing for variant Atomic on enabled arches- x86_64, ppc64le and aarch64. From traceback, it seems cause of failure is common for all arches.

From latest nightly (Fedora-Rawhide-20170426.n.0), traceback from pungi log [1] on x86_64 is:
File "/usr/lib/python2.7/site-packages/pungi/util.py", line 466, in failable
yield
File "/usr/lib/python2.7/site-packages/pungi/phases/ostree_installer.py", line 42, in process
self.worker(compose, variant, arch, config)
File "/usr/lib/python2.7/site-packages/pungi/phases/ostree_installer.py", line 61, in worker
task_id = self._run_ostree_cmd(compose, variant, arch, config, source_repos, output_dir, volid)
File "/usr/lib/python2.7/site-packages/pungi/phases/ostree_installer.py", line 178, in _run_ostree_cmd
% (output["task_id"], log_file))
RuntimeError: Runroot task failed: 19218947. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20170426.n.0/logs/x86_64/Atomic/ostree_installer-3/runroot.log for more details.

In runrootlog [2] log for x86_64, traceback from lorax is:
Traceback (most recent call last):
File "/usr/sbin/lorax", line 273, in <module>
main()
File "/usr/sbin/lorax", line 133, in main
remove_temp=True, verify=opts.verify)
File "/usr/lib/python3.6/site-packages/pylorax/init.py", line 329, in run
size=size)
File "/usr/lib/python3.6/site-packages/pylorax/treebuilder.py", line 224, in create_runtime
"Anaconda", size=size)
File "/usr/lib/python3.6/site-packages/pylorax/imgutils.py", line 113, in mkrootfsimg
mkext4img(rootdir, outfile, label=label, size=fssize)
File "/usr/lib/python3.6/site-packages/pylorax/imgutils.py", line 442, in mkext4img
mkfsargs=["-L", label, "-b", "4096", "-m", "0"], graft=graft)
File "/usr/lib/python3.6/site-packages/pylorax/imgutils.py", line 427, in mkfsimage
copytree(rootdir, mnt, preserve)
File "/usr/lib/python3.6/site-packages/pylorax/imgutils.py", line 238, in copytree
runcmd(cp)
File "/usr/lib/python3.6/site-packages/pylorax/executils.py", line 341, in runcmd
return execWithRedirect(cmd[0], cmd[1:], **kwargs)
File "/usr/lib/python3.6/site-packages/pylorax/executils.py", line 228, in execWithRedirect
env_add=env_add, reset_handlers=reset_handlers, reset_lang=reset_lang)[0]
File "/usr/lib/python3.6/site-packages/pylorax/executils.py", line 201, in _run_program
raise subprocess.CalledProcessError(proc.returncode, argv, output)
subprocess.CalledProcessError: Command '['cp', '-a', '/var/tmp/lorax.kd978bi0/installtree/.', '/var/tmp/lorax.imgutils.v3h12bwm']' returned non-zero exit status 1.

Problem is happening while copying files from /var/tmp/lorax.kd978bi0/installtree/. to created root filesystem. While debugging locally, found that size of created filesystem is 2GB which is default size provided by lorax. When I checked size of /var/tmp/lorax.kd978bi0/installtree/. (source for copy) , it is 2.7G .

I ran locally on both x86_64 and ppc64le box, lorax command with additional option --rootfs-size=3 (to specify size of root filesystem in GiB) and was able to run successfully. Running lorax with additional option --rootfs-size with required size in our compose script/config should fix the problem.

[1] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170426.n.0/logs/global/pungi.global.log
[2] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170426.n.0/logs/x86_64/Atomic/ostree_installer-3/runroot.log


If rootfs_size option is added to this block (and possibly others) it should be passed on to lorax running in the runroot.

Thanks @lsedlar , @mohanboddu and @ausil for getting this issue fixed. With this fix, nightly compose for Atomic variant in rawhide is building successfully for x86_64 and ppc64le arches since Fedora-Rawhide-20170429.n.0 ( https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170429.n.0/compose/Atomic/ ) . Still failing for aarch64 but it's a different issue.

Metadata Update from @sinnykumari:
- Issue status updated to: Closed (was: Open)

6 years ago

Metadata Update from @ausil:
- Issue close_status updated to: Fixed

6 years ago

Login to comment on this ticket.

Metadata