I believe what we are hitting here is #4084 @katec FYI

Is there anything we should do on the glibc side?

@fweimer Nothing needs to be changed. Takskotron check on whats available in fedora+updates repo, so until the multilib lands in the repos the same time the update that drags it in lands in those repos, taskotron cant do anything about them. Please file an issue at https://pagure.io/taskotron/task-rpmdeplint for better tracking.

Closing it since we cant do anything until the update lands in f27 repo.

@mohanboddu, I find your response confusing. I think the f27-testing compose is broken. Surely is a bug which needs fixing, with an immediate workaround to get f27-testing into a consistent state, and a long-term fix for whatever the underlying bug is.

The issue is back:

[fweimer@oldenburg ~]$ rsync rsync://ftp-stud.hs-esslingen.de/fedora/linux/updates/testing/27/x86_64/l/ | grep libcrypt
-rw-r--r--         45,052 2017/10/15 23:50:10 libcrypt-2.26-14.fc27.i686.rpm
-rw-r--r--         44,184 2017/10/15 23:50:00 libcrypt-2.26-14.fc27.x86_64.rpm
-rw-r--r--         36,800 2017/10/15 23:49:43 libcrypt-nss-2.26-14.fc27.x86_64.rpm
[fweimer@oldenburg ~]$ 

Another incident:

$ rsync rsync://ftp-stud.hs-esslingen.de/fedora/linux/updates/testing/28/Everything/x86_64/Packages/g/ | grep glibc-headers
-rw-r--r--        465,956 2018/05/18 17:21:33 glibc-headers-2.27-14.fc28.x86_64.rpm

glibc-headers.i686 is missing.

So, before we used pungi for GA releases and mash for updates and each had their own setup for this and it was confusing. Now we are only using pungi and it in turn uses python-multilib.

So, can you file a bug against python-multilib and see if we can get it fixed in that one place once and for all and then it should stay fixed hopefully.

https://bugzilla.redhat.com/show_bug.cgi?id=1582634

I don't think this is a python-multilib problem. It's caused by the process generating the updates composes. As I understand it, only actually updated packages get into the updates compose.

Looking at the example of glibc-headers, it's in nightly rawhide because libxcrypt-devel-4.0.1-1.fc29.i686 depends on it, not by any property of the actual glibc-headers package.

If there is no i686 package in the updates that would require glibc-headers, it will not be there.

I'm not sure how best to solve this.

A possible solution would be to use Pungi's prepopulate feature: it's a list of packages that should be included in the compose even if the configuration does not say so specifically. At GA time a package list would be generated from the compose that would say which packages are multilib (and on which variant and arch).

This file would be added to the config file used by Bodhi. If a package on the list is updated, it's multilib version would be pulled in.

This approach does not cover case when new multilib packages are added as updates. In such case the listing would have to be updated (either manually or via some automated process).

If this is an acceptable solution, I will need to write a script to create the listing.

Reopening, per: https://bugzilla.redhat.com/show_bug.cgi?id=1582634#c1

Why do we cannot ship a full, regular compose in updates-testing? Is it the metadata size?

Would it be possible to generate a regular compose and then drop all packages which are available elsewhere?

Comment 3 on compose-utils#70 links to what the package list would look like.

@mohanboddu will look into this issue.

The Fedora 29 GA compose has glibc-headers.i686, but the Fedora 29 updates compose lacks glibc-headers.i686.

This breaks updates for users if they have accidentally installed glibc-headers.i686. Please fix.

@lseldar I see in that example list there is debuginfo and debugsource, will pungi put those in the right place in the tree? Or would they get added to the main repos?

Otherwise, this seems ok to me, we could generate this for f29 and start trying to use it for updates-testing/updates.

@kevin Is it by any chance, we are filtering glibc and libgcc packages in pungi configs of normal composes but not in bodhi pungi configs?

Nope, those are different... glibc32, libgcc32 are special 32 bit packages we have in the koji x86_64 buildroot in order to build some 32 bit stuff. They never should be shipped to users.

glibc.i686 and libgcc.i686 are just the 32 bit versions of those and are shipped to all i686 users, and some x86_64 users to run 32bit binaries.

The problem here is that they are multilib in GA repo, so people can install them, but then in updates we only have the x86_64 one, so there's a mismatch and dnf complains and the user complaines, etc.

The debuginfo packages will still end up in debuginfo repo. Anything else would be a bug.

@humaton will talk to @mohanboddu about this ticket.

From our grooming discussion on #fedora-releng channel on Apr 12 2019

[15:46:08] <+nirik> so, on this thing... I think it's fixed right? we are using python-multilib and it should do the right thing no?
[15:46:18] <+mboddu> I think so
Proposal: Close the ticket and ask them to reopen if they find any issues