Learn more about these different git repos.
Other Git URLs
We are trying to build AtomicHost installer with f28 stable updates pushes. The build works, but then it fails copying from the work tree to the compose:
[OSTREE_INSTALLER] [INFO ] [FAIL] Ostree installer (variant AtomicHost, arch ppc64le) failed, but going on anyway. [OSTREE_INSTALLER] [INFO ] ERROR running command: cp -av /mnt/koji/compose/updates/Fedora-28-updates-testing-20180502.0/work/ppc64le/AtomicHost/ostree_installer/* /mnt/koji/compose/updates/Fedora-28-updates-testing-20180502.0/compose/AtomicHost/ppc64le/os/
See: https://pagure.io/dusty/failed-composes/issue/232
$ ls -la /mnt/koji/compose/updates/Fedora-28-updates-testing-20180502.0/work/ppc64le/AtomicHost/ostree_installer/ total 32 drwxrwxr-x. 5 root root 4096 May 2 12:39 . drwxr-xr-x. 4 apache apache 4096 May 2 12:19 .. drwxrwxr-x. 3 root root 4096 May 2 12:38 boot -rw-rw-r--. 1 root root 40 May 2 12:25 .discinfo -rw-r--r--. 1 root root 2532 Apr 23 17:03 Fedora-Legal-README.txt drwxrwxr-x. 2 root root 4096 May 2 12:38 images -rw-r--r--. 1 root root 1063 Apr 21 10:12 LICENSE drwxrwxr-x. 3 root root 4096 May 2 12:38 ppc -rw-rw-r--. 1 root root 291 May 2 12:39 .treeinfo
The issue seems to be it's doing a cp -av as apache and the ostree_installer dir is root owned, and it's unable to keep the exact ownership there.
# sudo -u apache cp -av ostree_installer/* /mnt/koji/compose/updates/Fedora-28-updates-testing-20180502.0/compose/AtomicHost/ppc64le/os/ 'ostree_installer/boot/grub/grub.cfg' -> '/mnt/koji/compose/updates/Fedora-28-updates-testing-20180502.0/compose/Atomic Host/ppc64le/os/boot/grub/grub.cfg' cp: failed to preserve ownership for '/mnt/koji/compose/updates/Fedora-28-updates-testing-20180502.0/compose/AtomicHost /ppc64le/os/boot/grub/grub.cfg': Permission denied ...
I'm not sure the best fix here. Perhaps pungi could always chown apache at the end of run_roots? Or could add a config option for what user it's running as and chown as that if it's not root?
Or switching the 'cp -av' to a 'cp -rv' would fix things probibly.
I can see both of the solutions working. We could chown the output to the current user running the compose (I don't see much need for configuration here), or just copy without preserving owner.
Proposed fix in #935.
Attempt no. 2: #936.
Commit dc557b8 fixes this issue
The fix is backported into pungi-4.1.24-2.fc27
this issue is not fixed: https://pagure.io/dusty/failed-composes/issue/309
Metadata Update from @dustymabe: - Issue status updated to: Open (was: Closed)
Yeah. I looked some more this weekend and couldn't see the problem.
I think we will need to add in some debugging (like the exact error code from the cp and all the stdout/stderror from it).
do we see this same error if we run it in stage? can we try that and instrument some debugging in there?
I'm not sure if we can run a full bodhi update/compose in staging currently...
@bowlofeggs can we run bodhi composes in stg? and are we composing atomichost installer images there?
Hi @kevin!
Bodhi composes were working in stg until approximately 2 months ago. For some reason, Pungi always fails there nowadays. I don't know enough about Pungi to debug it, but I would really appreciate it if someone with Pungi knowledge were able to get it working again. The Bodhi side of composing does fully work in stg.
The OSTree config has never worked in stg and I also don't know what that is. To work around that, there is an if statement in the Ansible template the excludes it from running there. It would be equally swell if someone could get that working too.
@bowlofeggs I would be happy to work with you on getting stage working. Any proposal how to get started?
@kevin Instead of calling cp we could copy the files with python code. There already is code for that. This way the error message should be much better. PR 948.
cp
@bowlofeggs The OSTree config has never worked in stg and I also don't know what that is. To work around that, there is an if statement in the Ansible template the excludes it from running there. It would be equally swell if someone could get that working too.
For this particular issue it's just the ostree_installer that is failing and I'm pretty sure we should be able to run that in stage without any issues (assuming pungi in general can work). Does that bit of info help?
On 05/15/2018 06:28 AM, Lubomír Sedlář wrote:
@bowlofeggs I wold be happy to work with you on getting stage working. Any proposal how to get started?
That would be fantastic. How about I'll file an infrastructure ticket that talks about the problems I'm seeing and I can paste logs in there and what not? We can use that as a medium to discuss. It might be a while before I have time to do it because things are $crazy right now, but I'm adding it to my task list.
That would be fantastic. How about I'll file an infrastructure ticket that talks about the problems I'm seeing and I can paste logs in there and what not?
Yes please!
Commit 2862ae0 relates to this ticket
pungi-4.1.24-4.fc27 contains a patch that should improve the traceback with more details about what's going wrong.
oops. I thought I updated this, but I did not. ;( I will update it as soon as todays pushes are done and we can see tomorrow how it goes or I can do another f28 stable if there's something to push...
It is now updated... the next pushes should use it.
I just started the push for today, lets see what happens.
The logs show that the problem is caused by files with permissions 700. Since we no longer try to preserve the owner, it should be sufficient to make the files readable to whole world. PR 962 might do the trick.
Commit 92b5ad2 relates to this ticket
I made a build 4.1.25-2 (there was an upstream release in the meantime) that includes a patch to make all files readable at the end of the runroot task. That should hopefully fix the problems.
new failure: https://pagure.io/dusty/failed-composes/issue/412
The issue is that the chmod masked exit code from the actual command, and thus the task was successful despite not producing the required data. Pungi 4.1.25-3 should fix that.
ok got 4.1.25-3 in place and worked around the lorax issue . We are still seeing some copy errors for aarch64 and x86_64 surprisingly ppc64le works: https://pagure.io/dusty/failed-composes/issue/419
4.1.25-3
aarch64
x86_64
ppc64le
Ah. This is not the efi directory itself, but files underneath it. We probably want a -R (--recursive) for the chmod.
+1, this has been a battle :)
facepalm … how did I not add that option the first time? PR #971
Commit 663a070 relates to this ticket
can we get a build of this so we can try it for the next f28-u-t run?
Yes you can, it's already running as pungi-4.1.25-4.
Fedora-28-updates-testing-20180606.0 had successful ostree installer. This looks finally fixed.
Metadata Update from @lsedlar: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.