#448 Improve installability for Fedora
Opened a year ago by mvadkert. Modified 5 months ago

This is a collection of issues and improvements for the installability test in Fedora.
We need to make this test gating in the future to be able to prevent broken builds entering Fedora.

This was orginally files as a follow up to #447, seems the issues was missed because the installability test is unstable in Fedora. I see it being stable in RHEL, so we need to investigate how to make it stable so the developer can rely on it.

Improve the test output and documentation, to be better understandable by packagers

From @zbyszek: The tests always fail. I tried to fix them, but there are very little docs, and the docs that are there are (at least for me) just incomprehensible. I asked in various places (e.g. https://pagure.io/fedora-project-config/issue/292), but that never went anywhere. But if you can make the tests return useful results, I would be happy to look at them in the future.

systemd journal should be saved to improve the investigation

From @zbyszek - It would also help a lot if the journal was saved. Ideally, as a standalone file produced by systemd-journal-remote. Having the messages in context and with metadata, and not just the console output, would it easier to diagnose failures.

Make the installability groups configurable

From @zbyszek - I'd like to have the installability of all packages testable, even if they conflivt. Just opting out of testing altogether is a workaround, not a solution. When this was discussed in the past, I asked for a way to divide packages into groups, so that subsets are installed separately. (Ideally, this grouping could be expressed using glob patterns. This would I won't again forget to update this list when adding another *-standalone package.)

Package examples which would benefit from this config: systemd, fedora-release, curl

This should include a set of assertions of the packages installed. Recently, podman broke with loosing a recommends - https://github.com/containers/podman/issues/21392. This was not caught by current version of installability -https://bodhi.fedoraproject.org/updates/FEDORA-2024-bdb021ff9e .

Installability should be prepared for dnf5

To prevent issues like #416 once dnf5 is here

MInimize test dependencies

For example copr outage should not cause the tool to fail, see #314

Installability does not work well for non-x86_64 packages

See for example this PR failing on installability for a aarch64 only build:
https://src.fedoraproject.org/rpms/widevine-installer/pull-request/1

The test should definitely be able to test aarch64 packages, but next to it should also fail reasonably if it cannot test on a certain architecture.

https://artifacts.dev.testing-farm.io/fbca15cc-853c-4f58-aedf-c5061f4b2dc8/

If the old package is broken, skip downgrading / upgrading to it

It happened before that the old package is broken, quite an expected state give the state of missing gating etc. The expectation is to gracefully skip downgrade/upgrade tests if old package is broken.

See https://gitlab.com/testing-farm/oculus/-/issues/18 for details.


The reason why installability works in RHEL is because it only tests subset of RPMs that are actually later shipped to customers. I.e. packages like "systemd-standalone-sysusers" are skipped (iirc).
Installability in RHEL relies on test composes to figure out which RPMs would actually appear in the next nightly, if the build passed gating. That's how the RPM filtering is implemented.
This probably wouldn't work in Fedora as (I assume) all RPMs are shipped.

There is no "exclude this RPM/subpackage from installability testing" option implemented in the test at the moment. I am pretty sure there was a RFE for it somewhere, but I cannot find it at the moment.

@msrb sounds like the test should just consume a simple configuration file from the repository so the user can tune it to ignore installation of certain packages for example.

Also I see the viewer not ported, so will try to also do that

Metadata Update from @mvadkert:
- Issue assigned to mvadkert

a year ago

Metadata Update from @mvadkert:
- Issue priority set to: Critical (was: Medium)
- Issue tagged with: installability

a year ago

@zbyszek thanks, that is exactly the input we need for this test. This configuration looks like for rpm-install-test which is not the one running for bodhi updates.

But please note that I'd like to have the installability of those packages tested too. Just opting out of testing altogether is a workaround, not a solution. When this was discussed in the past, I asked for a way to divide packages into groups, so that subsets are installed separately. (Ideally, this grouping could be expressed using glob patterns. This would I won't again forget to update this list when adding another *-standalone package.)

Another (important!) package which has a similar issue is fedora-release: it has a dozen subpackages with complex interdependencies, which cannot all be installed together.

@zbyszek Aren't those *-standalone subpackages conflicting with the main systemd package, which is always installed on the system where the tests run?

On "Improve the test output and documentation, to be better understandable by packagers" - I did https://github.com/fedora-ci/installability-pipeline/pull/25 a while back, and that is in production now. A pass looks like this and a failure looks like this. @zbyszek would you say that's an improvement?

A pass looks like this and a failure looks like this. @zbyszek would you say that's an improvement?

It does look pretty good. In the failure case, it seems obvious how to figure out what went wrong, it that's what you were asking.

Found another thing while researching my devconf talk today - https://pagure.io/fedora-ci/general/issue/487

I definitely agree with this statement:

Just opting out of testing altogether is a workaround, not a solution.

However, I think it would be better to have iterative improvement on this instead of waiting for the perfect solution. The current situation, where uninstallable packages are being pushed to stable, is really bad. Getting to a state where the vast majority of packages are gated on installability, and a handful of packages such as systemd and fedora-release are opted out, would be a huge improvement.

Log in to comment on this ticket.

Metadata