Hi,

I discovered that our rawhide image was not rebuilt since the f32 branching. And it seems that depends-on packages (even published on rawhide) have not been installed so built python-setuptools have not been installed by dependency. The rawhide images build has been fixed few hours ago.

For another PR on python-gear I see that the built package is used during rpm-test: https://fedora.softwarefactory-project.io/logs/9/9/b88e00e56d1176eb639861e06c3785cb0d4c9da3/check/rpm-test/271d712/artifacts/installed-packages.list
The tests.yml set it as required_packages. and it is made available on the system by https://fedora.softwarefactory-project.io/logs/9/9/b88e00e56d1176eb639861e06c3785cb0d4c9da3/check/rpm-test/271d712/ara-report/file/b6c28210-6577-4efa-ae04-b248fe5a5897/

So you said "the "Fedora CI" said that the tested RPMs are always installed". So just to be sure:
Should the rpm-test job install by default all rpms built by rpm-scratch-build of tested PR ?

Should the rpm-test job install by default all rpms built by rpm-scratch-build of tested PR ?

In my opinion, not necessarily. But it should make sure that if any packages that match the ones built by rpm-scratch-build of tested PR are installed, that the ones form the PR are. And fail hard if that doesn't work because of dependencies.

Pseudo code:

for rpm in scratchbuild:
    if rpm.arch == 'src':
        continue
    if package_is_installed(rpm.name):
        if not exactly_this_built_is_installed(rpm):
            try:
                reinstall_exactly_this_rpm(rpm)
            except ReinstallationFailed:
                die_horribly()

I did that implementation through https://pagure.io/fedora-zuul-jobs/pull-request/44 and validated by using depends-on in that PR https://src.fedoraproject.org/rpms/python-gear/pull-request/9.

The check-installed.sh run after the run to the embedded tests.yml playbook. I think that's good to make sure that we have the rpms at the right versions on the system, but I'm not sure if the script should try to update the installed packages to the built version because at that point the functional tests.yml test is already done and the updated packages won't be tested.

Should we simply raise the error as soon as we detect that the built version of package is not installed after the run of tests.yml (and remove the update step from the algo) ?

@churchyard what do you think ?

Looking, sorry for the delay.

but I'm not sure if the script should try to update the installed packages to the built version because at that point the functional tests.yml test is already done and the updated packages won't be tested.

Can we update things before we run the tests? I am a bit lost on why this happens afterwards.

The check-installed.sh script could run before the task that run the embedded tests/tests.yml. But as tests/tests.yml installs the required_packages then no packages to be tested will be installed on the test system before tests/tests.yml run. Then check-installed.sh run will be irrelevant.
Except if we force the installation of packages from the PR being tested at the beginning of the job.

My assumption with that rpm-tests job was that:
- Packages installation + functional tests are handled by the STI tests/tests.yml.
- The rpm-tests job will be really simple: install the rpm repos + run ansible tests/tests.yml.

But as tests/tests.yml installs the required_packages

I wonder why doesn't it prefer packages from zuul-build repo :(

sorry I don't get your point :(
The zuul-build repo is installed on the test node and if the package to be validated is listed in the required_packages it will be installed.

https://fedora.softwarefactory-project.io/logs/9/9/b88e00e56d1176eb639861e06c3785cb0d4c9da3/check/rpm-test/271d712/artifacts/installed-packages.list here we see python3-gear is installed from "zuul-build" repo.
And here we see it is in required_packages https://src.fedoraproject.org/rpms/python-gear/blob/master/f/tests/tests.yml#_11

The point is quite simple when I look away from all the implementation things:

"When a CI job runs on package pull request and the package is installed, it must be built and installed from the PR's code, otherwise we are not actually testing the PR."

Now when I look at all the logic here, installing all packages from the PR is probably the safest way to achieve this.

Ok. Is it fine for you If we merge this: https://pagure.io/fedora-zuul-jobs/pull-request/53 ?
This installs the zuul-built packages.

https://src.fedoraproject.org/rpms/python-gear/pull-request/29 is a success on Zuul (shouldn't be), and a failure on Jenkins.

I think it has to do with repo priorities or costs.

https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/blob/master/config/Dockerfiles/fedoraci-runner/virt-customize/virt-customize.sh#L71

priority=0

This means the package is installed from the test repo, even if the EVR is the identical (possibly even if lower) than the current package from Fedora EVR.

Hi,

https://pagure.io/zuul-distro-jobs/pull-request/66

Thanks for the investigation it seems to work as expected.
Feel free to close the issue if your are fine with rpm-test job.

Thank you.