#12474 No Fedora Budgie Atomic compose since 41.20241114.0
Closed: Fixed 13 days ago by siosm. Opened 3 months ago by siosm.

  • Describe the issue

The last successful Fedora Budgie Atomic compose is from 41.20241114.0.

When looking at the logs, they are cut in weird places:
- https://kojipkgs.fedoraproject.org/compose/updates/Fedora-41-updates-20241116.0/logs/x86_64/Onyx/ostree-4/
- https://kojipkgs.fedoraproject.org/compose/updates/Fedora-41-updates-20241116.0/logs/x86_64/Onyx/ostree-4/create-ostree-repo.log
- https://kojipkgs.fedoraproject.org/compose/updates/Fedora-41-updates-20241116.0/logs/x86_64/Onyx/ostree-4/runroot.log

And the failure is weird:

FAILED: BuildrootError: rm exited with status 1

For comparison, the unofficial builds have been successful so far, pointing to an issue specific to the Fedora infra: https://gitlab.com/fedora/ostree/ci-test/-/pipelines

  • When do you need this? (YYYY/MM/DD)

No specific deadline. As soon as possible.

  • When is this no longer needed or useful? (YYYY/MM/DD)

N/A

  • If we cannot complete your request, what is the impact?

Fedora Budgie Atomic 41 users are not receiving updates.


I see:

DEBUG util.py:463:  systemd-udev.post: Created symlink '/etc/systemd/system/multi-user.target.wants/systemd-homed.service' → '/usr/lib/systemd/system/systemd-homed.service'.
DEBUG util.py:463:  systemd-udev.post: Created symlink '/etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service' → '/usr/lib/systemd/system/systemd-homed-activate.service'.
DEBUG util.py:463:  systemd-udev.post: CreateTraceback (most recent call last):
DEBUG util.py:463:    File "/usr/bin/pungi-make-ostree", line 33, in <module>
DEBUG util.py:463:      sys.exit(load_entry_point('pungi==4.7.0', 'console_scripts', 'pungi-make-ostree')())
DEBUG util.py:463:               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
DEBUG util.py:463:    File "/usr/lib/python3.13/site-packages/pungi/ostree/__init__.py", line 202, in main
DEBUG util.py:463:      func()
DEBUG util.py:463:      ~~~~^^
DEBUG util.py:463:    File "/usr/lib/python3.13/site-packages/pungi/ostree/tree.py", line 155, in run
DEBUG util.py:463:      self._make_tree()
DEBUG util.py:463:      ~~~~~~~~~~~~~~~^^
DEBUG util.py:463:    File "/usr/lib/python3.13/site-packages/pungi/ostree/tree.py", line 62, in _make_tree
DEBUG util.py:463:      shortcuts.run(
DEBUG util.py:463:      ~~~~~~~~~~~~~^
DEBUG util.py:463:          cmd,
DEBUG util.py:463:          ^^^^
DEBUG util.py:463:      ...<3 lines>...
DEBUG util.py:463:          universal_newlines=True,
DEBUG util.py:463:          ^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG util.py:463:      )
DEBUG util.py:463:      ^
DEBUG util.py:463:    File "/usr/lib/python3.13/site-packages/kobo/shortcuts.py", line 347, in run
DEBUG util.py:463:      lines = proc.stdout.read(buffer_size)
DEBUG util.py:463:    File "<frozen codecs>", line 325, in decode
DEBUG util.py:463:  UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: invalid continuation byte
DEBUG util.py:610:  Child return code was: 1

Ah, I've seen this one a few times now. I'll file a pungi issue.

Metadata Update from @phsmoura:
- Issue tagged with: investigation, medium-gain, medium-trouble, ops

3 months ago

Metadata Update from @siosm:
- Issue status updated to: Open (was: Closed)

a month ago

It's probably something in that package set emitting non-UTF8 content in its %post.

But pungi really needs to learn to deal with this by probably using utf8 replacement in its subprocess handling.

So I poked around at this and came up with https://pagure.io/pungi/pull-request/1812 . wdyt ?

Ah, for the record, I see the fix @hlin tried - https://pagure.io/pungi/c/7d8f3b4b9b2cf65967b4d3f8dd249aec2e3cbbf8?branch=master - but I can also see why it didn't work. It effectively bypasses kobo and calls subprocess.Popen directly, and handles subprocess.Popen returning a non-zero exit code, but that's just not the failure path here. The failure path is that subprocess.Popen raises a UnicodeDecodeError.

We could catch that exception and move on, but I don't like that because it means we drop rather abruptly out of the process, and we will lose a lot of output. Using subprocess' ability to replace invalid characters seems much better.

OK, I think my fix is good, Onyx didn't fail in the latest Rawhide. I'll install the same pungi build on the stable compose host.

Huh. It failed again in today's updates compose, but clearly wasn't using the new pungi. Maybe I updated the wrong compose host. I'll check tomorrow...

duh, of course, updates composes run on bodhi:

2025-01-13 00:15:52 [INFO    ] Host: bodhi-backend01.iad2.fedoraproject.org
2025-01-13 00:15:52 [INFO    ] Pungi version: pungi-4.7.0-8.fc41.noarch

just need to update that host...I'll do that.

d'oh, and it still doesn't work, but I see why not. we're using something from pungi really within the koji task itself, this isn't the pungi on the compose host. The 'outer' pungi is 4.8.0-3.fc41 with the patch, but the 'inner' pungi is 4.8.0-1.fc41 without it:

DEBUG package_manager.py:295:  ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f41-build-56577093-6544438/root/', 'install', 'pungi', 'ostree', 'rpm-ostree', 'selinux-policy-targeted', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing']
...
DEBUG util.py:461:  Installing:
DEBUG util.py:461:   ostree                            x86_64 2025.1-1.fc41              build      621.4 KiB
DEBUG util.py:461:   pungi                             noarch 4.8.0-1.fc41               build      313.0 KiB

I don't think there's an easy way to get it to use a newer build for the 'inner' pungi besides getting the update pushed stable, so if people could karma it that'd be great - https://bodhi.fedoraproject.org/updates/FEDORA-2025-9fded5cd72

still needs one more karma or this won't be fixed for several days.

Huh. Weirdly, we got an Onyx in the most recent updates compose, but it didn't have the Pungi fix yet. Possibly someone fixed whatever package scriptlet is generating non-unicode output? But hey, it's good news.

alright. Should we close this then? Or wait a bit more?

This is resolved as far as I can see. Thanks for the fixes!

Metadata Update from @siosm:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

13 days ago

Log in to comment on this ticket.

Metadata
Boards 1
Ops Status: Backlog
Related Pull Requests
  • #1809 Merged 24 days ago