#8330 no f30 silverblue updates: ostree composes started by bodhi are failing
Closed: Fixed 2 years ago by dustymabe. Opened 3 years ago by dustymabe.

  • Describe the issue

The ostree composes seem to be failing with an error:

2019-03-12 16:00:50 [ERROR   ] [FAIL] Ostree (variant Everything, arch x86_64) failed, but going on anyway.
2019-03-12 16:00:50 [ERROR   ] 'ascii' codec can't decode byte 0xc2 in position 6271: ordinal not in range(128)

An example failed composes issue for this: https://pagure.io/dusty/failed-composes/issue/1642

  • When do you need this? (YYYY/MM/DD)

ASAP so people on Fedora SB 30 can get updates

  • When is this no longer needed or useful? (YYYY/MM/DD)

F30 EOL

  • If we cannot complete your request, what is the impact?

Users of Fedora SB move to something else (possibly not Fedora).


The ostree composes weren't failing in the normal pungi runs before release (i.e. not run through bodhi). Mohan pointed out one difference between those setups:

09:55:46*       lsedlar | mboddu, is there the same python version as well?
09:56:29        +mboddu | lsedlar: Nope
09:57:06*       +mboddu | bodhi-backend01.phx2.fedoraproject.org has python3-3.7.3-1.fc29.x86_64
09:57:19*       +mboddu | where as the compose-x86-01.phx2.fedoraproject.org has
                        | python3-3.7.2-5.fc29.x86_64
09:58:01        +mboddu | lsedlar: bodhi-bakend01 has the latest version compared to both of them

One thing I'm a bit confused on is why Pungi isn't doing UTF-8 for subprocesses by default. Something like this?

diff --git a/pungi/wrappers/kojiwrapper.py b/pungi/wrappers/kojiwrapper.py
index 3cd9ac9..02855bb 100644
--- a/pungi/wrappers/kojiwrapper.py
+++ b/pungi/wrappers/kojiwrapper.py
@@ -147,7 +147,8 @@ class KojiWrapper(object):
         task_id = None
         with self.get_koji_cmd_env() as env:
             retcode, output = run(command, can_fail=True, logfile=log_file,
-                                  show_cmd=True, env=env, universal_newlines=True)
+                                  show_cmd=True, env=env, universal_newlines=True,
+                                  encoding='UTF-8')
         if "--task-id" in command:
             first_line = output.splitlines()[0]
             if re.match(r'^\d+$', first_line):

So reading the backstory here in https://pagure.io/dusty/failed-composes/issue/1642, it seems like it's getting thrown off by some specific Unicode characters rpm-ostree emits. And some changes in Unicode handling in 3.7.3 is making this reliably fail. Seems like we should be able to just tweak pungi to decode rpm-ostree output using utf-8?

In the worst case we could tweak rpm-ostree for this, but... it's utf-8. I'm sure rpm-ostree isn't the only app we'd want to run in a runroot that outputs utf-8. :)

This part also looks suspicious:

bin/pungi-koji
9:import locale
29:# force C locales
30:locale.setlocale(locale.LC_ALL, "C")

I think that should probably be C.UTF-8 nowadays.

This kobo function (used in a few places in pungi) also looks suspicious:

def save_to_file(filename, text, append=False, mode=0o644):
    """Save text to a file."""
    if append and os.path.exists(filename):
        fd = os.open(filename, os.O_RDWR | os.O_APPEND, mode)
    else:
        fd = os.open(filename, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, mode)
    if not isinstance(text, bytes):
        text = bytes(text, encoding=locale.getpreferredencoding())
    os.write(fd, text)
    os.close(fd)

Having one function to create both binary data and text seems odd...given the use cases I'd just assume any bytes() instances are UTF-8?

This bug is pretty urgent IMO because we shipped people an operating system and aren't giving them security updates - anything I can do to help please let me know. If truly necessary we can try to strip out the UTF-8 usage in rpm-ostree but...can a Pungi maintainer comment on https://pagure.io/releng/issue/8330#comment-569167 ?

I downgraded bodhi-backend01.phx2.fp.o to python3-3.7.2.-5, and started f30-updates and f30-updates-testing push. Lets see what happens.

I downgraded bodhi-backend01.phx2.fp.o to python3-3.7.2.-5, and started f30-updates and f30-updates-testing push. Lets see what happens.

Thanks @mohanboddu.

He did this so we can see if we can get a new update out today. The pungi maintainers would only be able to get back to us at the earliest tomorrow morning and we'd like to get an update out sooner than later. Since the attempt at a workaround here is a downgrade this is only a temporary bandaid so we'll have to dig in and possibly implement the suggestions from @walters and @jlebon soon.

Okay, I am still seeing this after downgrading python3 in the latest bodhi compose

OK, testing a temporary hack in https://src.fedoraproject.org/rpms/rpm-ostree/pull-request/32.

cool - once we get a build would we need to use a buildroot override?

OK, new rpm-ostree in buildroot with the patch above. Can someone retry the updates & updates-testing push?

/cc @kevin

f30 updates-testing away. (if that works we can push updates too, but best to just test with updates-testing)

It seems to have worked. If we get a f30-updates I think we can be happy... and look for longer term fix in pungi.

f30-updates succeeded as well. Thanks all!

Both updates and testing ref has not yet synced to production ostree/repo/refs/

@walters
One thing I'm a bit confused on is why Pungi isn't doing UTF-8 for subprocesses by default.

open PR in https://pagure.io/pungi/pull-request/1185

@sinnykumari
Both updates and testing ref has not yet synced to production ostree/repo/refs/

right - we'll need to look into getting those synced out and also alias the stable ref to the updates ref so that they are one in the same

ok - we have an updates ref now - i'm running a quick update on my laptop to make sure it's not obviously broken before I alias the stable ref to it.

it looked good. I got the stable ref aliased over and everyone should be able to update now 🌈

Metadata Update from @dustymabe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

actually i'll re-open this as we applied a bandaid (patch to rpm-ostree) but the real fix is in pungi. I'll re-open until we confirm it's fixed (open PR is here)

Metadata Update from @dustymabe:
- Issue status updated to: Open (was: Closed)

3 years ago

we can close this now since it's been a while

Metadata Update from @dustymabe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata