Just FTR, now I am checking also the other tests from the Bodhi updates and the situation is completely similar with rpmdeplint as well with installability test. The situation is also worsened, that the results on the installability and rpminspect test pages are sorted.

Actually, we want to link users to only one place, to Testing Farm UI, which will be unified between all CI systems they use it (RHEL CI, Fedora CI, Zuul CI, Automotive CI, etc.)

In this case it is this:
http://artifacts.dev.testing-farm.io/7fa4436b-fc89-4e3b-9632-09f6f11def02/

At least that way we have one place to fix to solve this problem for all.

http://artifacts.dev.testing-farm.io/7fa4436b-fc89-4e3b-9632-09f6f11def02/

1) It does not provide any logs at all
2) It again tries to sort the results which is not of benefit

Please take it as an early feedback ;)

@vondruch

Because to be able to reproduce CI failure locally, I am really not interested in details that something was executed somewhere in container. This is noise for CI user.

What exactly do you mean when you say "reproduce a CI failure locally"? Would the answer be the same for both tests that you own (tmt/tier0.functional -- basically tests under your control) and "generic" tests like rpminspect?

@msrb Sorry, I am not sure I understand your question. But I guest this is the part of the problem. You know the system and you have very precise questions. I don't know the system and it does not provide enough easily available information for me to even being able to learn.

If you question is that if I'd like to reproduce the CI issues via TMT, then the answer is "it depends". I'd like to be able to understand that there is used TMT and I'd also like to understand that TMT is executing some subcommands. To help me understand, I'd like to see the log containing the information.

BTW I have provided the Bodhi update for the reference, so we can talk in the context of that errata.

It seems that I am not the only one who struggles:

https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/message/XAF5JUX3X4GLFCUOSYFMZNHUKH3JWN6T/

I think that @adamwill made similar experience to myself.

Right. I ran into an issue with a CI test and wanted to find the code that seemed to me to be not behaving optimally and see if I could suggest a change to it, but was ultimately unable to find that code. Not sure specifically how my experience could be improved, but there it is. If I do manage to find it and have any more specific suggestions, I'll update this ticket.

I think there are multiple problems here.

Let's start with the most straightforward one:

• I have my own tests in dist-git and I want to be able to debug CI failures locally

I think this is where tmt truly shines. The assumption here is that you already know how to run your tests locally, via tmt (because you added the tests and the tmt metadata).
However, if the same tests fail in CI, then you probably want to know how to replicate that CI environment on your laptop so you can debug the problem. Basically, you know your tests, you know how to run them locally, but you need to be able to reproduce failures that only happen in CI on your laptop.

I can see at least 2 problems here:

• Testing Farm is not just tmt-running your tests (as you do locally), it also prepares the environment in a certain way -- it prepares a local YUM repository with your builds downloaded from Koji and then it tries to install them
• currently it's not clear what compose/image is used to run the tests, and whether it is dnf-updated before running the tests

I think the env-prepare functionality should be moved to tmt (somehow, plugin maybe?), so people can actually prepare the environment the same way locally.
And the information on which compose/image was used to run the tests in CI should be visible somewhere.

Ideally, there should be a single tmt command somewhere in CI that people could copy/paste, if they wanted to reproduce the CI run on their laptops.

Hi,

On Mon, Jan 17, 2022 at 1:26 PM Michal Srb pagure@pagure.io wrote:

msrb added a new comment to an issue you are following:
``
I think there are multiple problems here.

Let's start with the most straightforward one:

• I have my own tests in dist-git and I want to be able to debug CI
  failures locally

I think this is where tmt truly shines. The assumption here is that you
already know how to run your tests locally, via tmt (because you added th=
e
tests and the tmt metadata).
However, if the same tests fail in CI, then you probably want to know how
to replicate that CI environment on your laptop so you can debug the
problem. Basically, you know your tests, you know how to run them locally=
,
but you need to be able to reproduce failures that only happen in CI on
your laptop.

+1

I can see at least 2 problems here:

• Testing Farm is not just tmt-running your tests (as you do locally), it
  also prepares the environment in a certain way -- it prepares a local YUM
  repository with your builds downloaded from Koji and then it tries to
  install them
  

Yes, we are working on this, and it will be resolved. tmt will gradually
takeover this responsibility so the user can completely reproduce what
testing farm does.

• currently it's not clear what compose/image is used to run the tests,
  and whether it is dnf-updated before running the tests
  

Right, we use vanilla Fedora images in AWS, they can be mapped to their
QCOW2 counterparts for local tmt execution.

Moreover, all the playbooks
https://gitlab.com/testing-farm/infrastructure/-/tree/testing-farm/ranch/p= ublic/citool-config/guest-setup
we run now in Testing Farm, will move under Fedora CI, where the playbooks
can be maintained by the community. The same will apply to all our other
users (CentOS Stream CI, Fedora CI, Packit, ...). Then this all will be
without our secret sauce, making things nice and generic, leaving the test
environment setup playbooks nicely coupled with the CI system, easily
consumable by the end user, i.e. end user will have one command to run
EXACTLY the same thing as CI does =F0=9F=9A=80

I think the env-prepare functionality should be moved to tmt (somehow,
plugin maybe?), so people can actually prepare the environment the same w=
ay
locally.
And the information on which compose/image was used to run the tests in C=
I
should be visible somewhere.

Ideally, there should be a single tmt command somewhere in CI that people
could copy/paste, if they wanted to reproduce the CI run on their laptops=
.

Agreed, this is exactly where we are heading (and we have weekly meetings
with @psss @mprchlik and @lzachar about it). It will takes us a bit of time
though, as moving these parts to tmt will require some significant changes.
But I am glad we completely see the ideal future the same way!

Best regards,
/M

``

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-ci/general/issue/311

--=20
Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE
IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252
Remote Czech Republic :: Red Hat Czech s.r.o

http://artifacts.dev.testing-farm.io/7fa4436b-fc89-4e3b-9632-09f6f11def02/

1) It does not provide any logs at all
2) It again tries to sort the results which is not of benefit

Please take it as an early feedback ;)

@msrb Can we start with these ^^ easy items and then maybe continue with the items you describe?

Actually, the "artifacts" service is WIP, but the suffers the same issues. I assume that it is produced by the similar means.

[1] https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpminspect-pipeline/job/master/67154/testReport/(root)/tests/

@vondruch yes, definitely a good start, we plan to do it soon, it was delayed a quarter now :(

It is https://issues.redhat.com/browse/TFT-641, I am adding sorting there

@vondruch I think is test-specific, but we are going back to rpminspect here, so could you please confirm that this is actually what would make your life easier when dealing with rpminspect in CI?

http://artifacts.dev.testing-farm.io/6f79208a-5d55-4974-a251-a67cad191e74/work-rpminspectDG3Psr/

Let me record my investigation:

So there is some directory, looking at the initial content, there is log.txt, which grabbed my attention. run.yaml? Hm, to small and yaml? Probably not of interest. tmt? I am not really into tmt. rpminspec/? The content inside should be probably numbered, because again, the sorting is not useful.

Back to the log.txt. Trying to wrap my head around. I came to the line 09:53:12 out: rpminspect version: 1.9-0.1.202112231810git.fc36 (with data package: 1.8-0.1.202201141656git.fc36), which looks to be the start of the rpminspect. But yet again, looking above the line, I can't see anything like rpminspect ..... command which I could and use. Everything is obfuscated by some container.

Moving further, the log looks somehow longer then I'd expected. Why it does not end right at the 09:53:12 Check result of '/license'. line? Are there some "independet" rpminspect steps? Should I run rpminspect gazillion times when preparing my package?

So back closer to the top, to the lines 09:35:36 Write file '/var/ARTIFACTS/work-rpminspectDG3Psr/rpminspect/execute/data/license/metadata.yaml'. Hm, it seems there is going on some configuration. But where should I take these files? May be I should really take look at the run.yaml? Ok, that was a fail. The only thing of interest could be the - /rpminspect plan, but where the heck this is coming from? No idea.

Lets take a look at tmt-run.log. Hm, nothing there, again some steps I saw already elsewhere, but no idea about basically anything, but at least there is visible fail /badfuncs. So there is some issue.

Maybe I should really start to think about installing tmt, but I don't really want to, I'd probably need to install it into my mock, or to my system. Dunno, checking and it has just 2.3 MB installed size and 5 python dependencies which is not bad after all. But I still don't want. So lets check the rpminspec/ folder again. Maybe prepare? Useless ... provision? Hm, container again, but otherwise useless. What is in execute. step.yaml - useless. results.yaml - one fail, I know that. Useless. tmt-file-submit, FF does not recognize the file type, I'd need to download that file. But 175 B? Probably useless. default/reboot_template again, I would need to download it and ?!? data? Hm, sorted content again, but I still think the content should be numbered to follow the order of execution. What was the failure? (luckily this time I have the fail /badfuncs note above).

So lets get into rpminspect/execute/data/badfuncs/. metadata.yaml? Hm, that can't be the thing. output.txt, yes, I saw it elsewhere, but again, there is not the freaking rpminspect command used to execute this test case.

Should I go back to the log.txt? I've been there and I still don't know what I need to do to reproduce the issue locally.

And this is the end. I don't want to go any further.

Actually, now I remember that I was told that the /diagnostics contains the rpminspect command, but should I trust it seeing the rest of the log?

@vondruch thanks for the thorough investigation. I did not expect you to like it -- I just wanted to confirm that we are on the same page here.

Redirected from https://pagure.io/fedora-infrastructure/issue/10469

Take this Rawhide update as an example:
https://bodhi.fedoraproject.org/updates/FEDORA-2021-18cf77c6ca

When I first submitted the update and saw the failures, I don't think I could click any of the red rows under the Automated Tests tab. Actually, I recall, there was nothing clickable under that tab.

However, right now, I can click on a red row, and I am presented with:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/95323/testReport/(root)/tests/

[ insert everything that @vondruch and @adamwill have already said ]

I was able to finally locate and fix the problem after a lot of stressful guess work that took several days, but isn't this what CI is supposed to help me avoid?

I want an easy to way to see why the CI failed -- the failing test, the logs, the commands run, etc..

I wish the CI system was easy to manoeuvre as our build system. Here are the corresponding pages for the build:
https://koji.fedoraproject.org/koji/taskinfo?taskID=79777668
https://koji.fedoraproject.org/koji/buildinfo?buildID=1865498

I don't understand every little nuance on those pages, but when something goes wrong, I know where to look, and usually I can make out what went wrong. I think the only exception is when the infrastructure behind the build system actually fails, because that leads to some odd errors, but the fact that something is odd isn't hard to make out.

So, that's roughly what I want the user experience of the CI to be like.

Otherwise, right now, I am left considering removing the CI tests from any package that I own. A hard to use CI system seems just as worse as not having any CI.

However, right now, I can click on a red row, and I am presented with:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/95323/testReport/(root)/tests/

Did Pagure fail to mark that URL as a hyperlink? Maybe we shouldn't have brackets in URLs?

@rishi Thanks for the comment.

Your problem is... interesting. I think it's different than @vondruch's problem as, in your case, CI is simply running "your" tests that you have in rpms/toolbox.
There are problems even with running people's own tests in CI, but I'd assume that you'd be familiar with output of your tests, and that you already know how to run them locally.

However, and I really did not see this coming, tests for your package were actually added by the package co-maintainer. So, yeah, they are "your" tests, but you're not really familiar with them. And I can see how debugging them (with added CI complexity) can be an absolute nightmare.

We cannot really fix the output of such tests, as that's up to test owners. But I believe once the problems that I described in my comment above are solved, it should be clear how to run those tests in CI-like environment on your laptop, which should (I hope) make the debugging easier.

One thing maybe that would help is to use tmt for wrapping around the tests instead of STI which:

• makes it possible to add test metadata to the test together with description so sharing of that knowledge is easier
• running tests from localhost should be a lot easier

Some success story in:
https://cockpit-project.org/blog/fmf-unified-testing.html

Let's compare running the tests to building the package.

Most of the lines that show up in Koji's build.log are emitted by the package's own build -- the same build that's part of it's upstream code base. However, they are still different from the lines an upstream contributor might be seeing, because of different toolchains and build flags, downstream patches, etc..

Then the lines in root.log are a bit more downstream specific, because the upstream contributor might not have the latest downstream builds, using a different distribution, building the entire dependency tree from source (eg., a lot of GNOME folks do this), using a language-specific delivery mechanism (eg., Go, Python, etc.). Even though at an abstract level all the dependencies were defined upstream and should be known to those contributors.

In short, even though the build and it's dependencies are owned by the upstream project, and even when there's upstream CI, downstream builds do break. Sometimes they break in non-obvious ways, and it's really useful to be able to easily see the failure.

The same applies for the CI.

In this case, the CI failed because a Go package used by Toolbox had abruptly changed it's behaviour. The change wasn't detected by the upstream CI, because it wasn't using the latest version of the dependency, but Fedora was. So, when the downstream CI fails, it's really not clear why it failed. Running it locally wouldn't be convenient either, because I might not have the latest packages installed locally.

Imagine how difficult it would be if Koji made it very difficult to get to build.log, and the suggestion was to rebuild it locally using Mock to figure out the failure. There are simply too many moving pieces.

The output of my tests is different for each run, and I really need to be able to see the output when it was run as part of the CI. Otherwise, how do I fix the failure?

By the way, the button to waive the CI failure doesn't seem to work either, but I was told that it's a Bodhi problem.

@mvadkert oh, right. In my imaginary universe, STI is already dead and youngsters don't even know what that was :D

@rishi Yes, the situation currently is:

• maintainer building a package in Koji, but redirecting some of the output to a file (perfectly fine thing to do, at least in CI)
• Koji only cares about stdout/stderr of the build and doesn't know/ignore the workdir where the file with extra output was created
• maintainer or someone else trying to debug an issue with the build, but only the stdout/stderr from the build is available

This issue should go away when the issue that @mvadkert mentioned is resolved (maybe the link is not publicly accessible -- sorry about that).

In your case, CI would give you this link: http://artifacts.dev.testing-farm.io/54dc1e22-ecd1-4de6-b1a4-8514b85e35f9/work-tests.yml9muORY/

However, as @vondruch confirmed, for generic tests, simply exposing the workdir probably won't be sufficient. There is more work on that front.

In your case, CI would give you this link:
http://artifacts.dev.testing-farm.io/54dc1e22-ecd1-4de6-b1a4-8514b85e35f9/work-tests.yml9muORY/

Yes, having easy access to links like that will already be a big improvement. :)

@rishi Thanks for the comment.

Your problem is... interesting. I think it's different than @vondruch's problem

I don't see @rishi 's description of his experiences any different to my. I think it is perfectly within the context I am describing.

BTW since TMT is mentioned here and there, I have looked again at the log.txt from your example and not surprisingly, there is not visible how to execute tmt. The closest thing I can identify in the log is:

podman exec tmt-var-ARTIFACTS-work-rpminspectDG3Psr-rpminspect sh -c export PREVIOUS_TAG=f36-updates TASK_ID=79796333 RPMINSPECT_PROFILE_NAME=rawhide REPOSITORY_URL=https://src.fedoraproject.org/rpms/ruby CONFIG_BRANCH=rawhide DEFAULT_RELEASE_STRING=fc36 GIT_COMMIT=5d3688479ab299cbd5657f9a503463a04d963d65 DEBUG=False; cat /etc/os-release

but sorry, that is not the tmt command I'd use. I don't want to explore any container to execute tmt. Even if I wanted, there is no trace to the source of the container.

The difference, as I see it, is that @rishi is asking to see "his" logs that are currently hidden and not exposed.
Since CI is simply running his tests, everything that he finds in those logs should be familiar to him and don't need any explanation (no matter how ugly those logs might be, all the output is under his control -- CI doesn't understand those tests anyway). It's his playground, and he needs visibility and reproducibility.

rishi's problems are (my POV):

• hidden log files that were generated while the tests were running
• in case that his tests are working just fine for him locally, then being able to reproduce the CI-like environment on his laptop could be tremendously useful. So he can easily debug why his tests are not working in CI

I think the problem with rpminspect is slightly different:

• logs/output cannot be ugly, because it's not people's own tests
• I don't think that reproducing rpminspect CI-like environment on people's laptop is that useful. If CI tells you that you are using "badfuncs", then if you reproduce the exact same environment locally (same NVR, everyting...), you will get the same result: your build is using badfuncs
• if rpminspect in CI tells you that there is something wrong with your build, then you probably don't want to debug rpminspect itself (unlike when you run your own tests and they fail)

The way I see it:

• generic tests like rpminspect should provide output that clearly tells people what the problem is and ideally how to fix it (or at least where to get help)
  • including information about what version generated the output, etc.
• once people understand what rpminspect is telling them, they probably want to prepare the fix, mock-build it locally and quickly check if their new build really fixes the problem (without Koji->CI round trip) -- therefore they need to know how to quickly run rpminspect on their own laptop, on their own builds (not to strictly replicate exactly what CI is doing).

If you take the current rpminspect pipeline, you can reproduce exactly the same run that happen in CI. But you cannot deviate from it (you cannot run it on your own mock-build for example -- the input is strictly a Koji task id).

BTW, we added some preview of at least workdir and tmt repro steps:

http://artifacts.dev.testing-farm.io/6f1b322b-e7e6-43a6-b3c9-0b936c919cdc/

BTW, we added some preview of at least workdir and tmt repro steps:
http://artifacts.dev.testing-farm.io/6f1b322b-e7e6-43a6-b3c9-0b936c919cdc/

@vondruch & others. Could you please take a look and share your honest feedback? This result is for rpminspect.

Thanks ;)

@msrb and by the way I completely agree with your points, we should differentiate the tests the user owns and tests which are generic, provided, and maintained by the CI system owners (rpmdeplint/rpminspect/installability).

@msrb and by the way I completely agree with your points, we should differentiate the tests the user owns and tests which are generic, provided, and maintained by the CI system owners

First of all, I completely disagree. @rishi might have slight advantage understanding the evolution of hist test suite, but for 3rd party viewer, there is no difference. I have to be able to understand all the test suites and my test suites must be understandable to others.

http://artifacts.dev.testing-farm.io/6f1b322b-e7e6-43a6-b3c9-0b936c919cdc/

Just a few notes as I am trying to understand what is going on.

• The log on the top is left somehow out of context
• I am not sure why the listed commands are not followed by their output.
• I don't understand the purpose of git -C testcode show-ref and it'd be nice if there is nothing like git -C testcode checkout -b df6796e0. Of course, from the logs, it should be possible to deduce that the default branch was at df6796e0 commit.
• Looking at the cloned repository and at the rpminspect.fmf, I don't see where I would get the /usr/local/bin/rpminspect_runner.sh. I am pretty sure I don't have such file on my system and I'm quite sure I would not love to have it.

This is the place where you got me lost.

BTW this better reveals, that each of the rpminspect test cases are executed separately, which is good. OTOH, executing each task separately diverge from the basic design of rpminspect, where I'd expect to execute everything in single sweep. This in turn makes understanding of logs more difficult, IMO

@msrb and by the way I completely agree with your points, we should differentiate the tests the user owns and tests which are generic, provided, and maintained by the CI system owners

First of all, I completely disagree. @rishi might have slight advantage understanding the evolution of hist test suite, but for 3rd party viewer, there is no difference. I have to be able to understand all the test suites and my test suites must be understandable to others.

Well, sure, but as there were certain steps taken to make the test run reasonably in connection to tmt (which of course might be improved), what is missing is propper description of the test, so you can easily discover the documentation yourself. That is what is tmt about.

tmt plan show

Should show reasonable documentation (and it does not currently).

That should help with contributing also to a generic test.

http://artifacts.dev.testing-farm.io/6f1b322b-e7e6-43a6-b3c9-0b936c919cdc/

Just a few notes as I am trying to understand what is going on.

• The log on the top is left somehow out of context

Yep, it is should be explained it is thetmt test plan running directory, there should be a link to https://docs.testing-farm.io explaining what is it (we are working on it).

• I am not sure why the listed commands are not followed by their output.

Because there are not the commands which run, this is the reproducer via tmt, the wrapper tool which runs the tests.

I agree tmt should display the commands it is running in the output:

https://github.com/psss/tmt/issues/1031

• I don't understand the purpose of git -C testcode show-ref and it'd be nice if there is nothing like git -C testcode checkout -b df6796e0. Of course, from the logs, it should be possible to deduce that the default branch was at df6796e0 commit.

Yep, that is fixed now, it was an unneeded command ...

• Looking at the cloned repository and at the rpminspect.fmf, I don't see where I would get the /usr/local/bin/rpminspect_runner.sh. I am pretty sure I don't have such file on my system and I'm quite sure I would not love to have it.

It is a command run inside the container:

provision:
    how: container
    # source: https://github.com/fedora-ci/rpminspect-image
    image: quay.io/fedoraci/rpminspect:d1a73fb

Again, test description should point to the container which should contain some documentation.

This is the place where you got me lost.

Frankly, what is the reason for the rpminspect container? Why is it not just simple dnf install rpminspect. Please drop that container. Please drop every container used in such way. They just obfuscate the output.

@vondruch

BTW this better reveals, that each of the rpminspect test cases are executed separately, which is good. OTOH, executing each task separately diverge from the basic design of rpminspect, where I'd expect to execute everything in single sweep. This in turn makes understanding of logs more difficult, IMO

You're absolutely right. To make things even worse, we actually run all tasks at the beginning and then pretend that we run the rest, but we only return cached results.
We do that because rpminspect can cache and reuse downloaded RPMs between runs, but it needs to unpack those RPMs every time. Which wouldn't be a problem for most components, but few heavyweights (IIRC: libreoffice, kernel, gcc, ...) would never reach the finish line.

Frankly, what is the reason for the rpminspect container? Why is it not just simple dnf install rpminspect. Please drop that container. Please drop every container used in such way. They just obfuscate the output.

That container serves a purpose and it really isn't the problem here. We install rpminspect from COPR, so it is very close to what's currently in rpminspect upstream master branch. However, if we just dnf-install the tool from COPR every time we want to run it (tests run in throwaway VMs/env), we might end up with CI that breaks randomly, depending on the state of the COPR repo.
We also do not rely on stable rpminspect in Fedora as it can be quite old (latest rpminspect in Fedora is ~3 months old).
With containers, we can update rpminspect "on-demand" and still have control over what's running in production. If the latest COPR build is broken, we simply won't promote it to production.

Now, why is it everything so weird?

We work with stdout. And if we don't express in tmt that rpminspect has multiple tasks, we would only get a single wall of text containing output of all tasks, and only a single "result", i.e. "rpminspect failed, one or more tasks somewhere, go grep that huge output". So we workaround this problem by pretending that we run each task separately -- to capture output for each task separately.

The question for me is: what are the benefits of wrapping tests like rpminspect in tmt? I think that the benefits of tmt for standard functional/dist-git tests are clear, but for generic tests not so much.
In any case, tmt alone for generic tests could work. You can always wrap the test in tmt, express dependencies, call tmt command and let it provision a VM/container for you, and then at the end of the day just do some post-processing on the output of your generic tests. However, then the question is how much control do you have over the output. And testing farm is/was pretty opinionated in this regard.
In standard CI, we would just run rpminspect once, let it produce some machine-parsable output and the do some post-processing to make sure that the output is consumable.

I guess I am trying to figure out where the line should be, what is the responsibility of tmt, how the testing farm fits in, and how generic tests should integrate with the two.

Is testing farm+tmt meant to be something like travis-ci and .travis.yml?

In standard CI, we would just run rpminspect once, let it produce some machine-parsable output and the do some post-processing to make sure that the output is consumable.

I believe we should maybe revisit this, tmt should provide a way how the test can create it's own results.yml (with nice subresults) instead of forcing such tools to run in a non standard way :( causing just confusion ... then we have cleared rpminspect execution, no wrapper script.

Is testing farm+tmt meant to be something like travis-ci and .travis.yml?

Yeah, I guess that should go to some FAQ, and agree on what is the answer here. I believe tmt's vision is more than that, and it is really about test observability and easy localhost execution, but it definitely smell in some occasions just like another CI format :)

That container serves a purpose and it really isn't the problem here. We install rpminspect from COPR, so it is very close to what's currently in rpminspect upstream master branch. However, if we just dnf-install the tool from COPR every time we want to run it (tests run in throwaway VMs/env), we might end up with CI that breaks randomly, depending on the state of the COPR repo.

A broken rpminspect because the copr build was broken is something a packager with average experience could understand and relate to (and observe locally with the same copr repo enabled). A non-transparent container with wrapper scripts in /usr/local/bin? Not so much.

That container serves a purpose and it really isn't the problem here. We install rpminspect from COPR, so it is very close to what's currently in rpminspect upstream master branch. However, if we just dnf-install the tool from COPR every time we want to run it (tests run in throwaway VMs/env), we might end up with CI that breaks randomly, depending on the state of the COPR repo.

A broken rpminspect because the copr build was broken is something a packager with average experience could understand and relate to (and observe locally with the same copr repo enabled). A non-transparent container with wrapper scripts in /usr/local/bin? Not so much.

Is the issue the fact that it runs in a container, or the overall obfuscation of what happens inside that container?

I hope for the latter

@msrb do you think that idea behind letting the tmt test generate results.yaml could resolve this? then we can run rpminspect as the user would, the only magic would be at the parser of results which would translate it to tmt's results.yaml?

I hope for the latter

@msrb do you think that idea behind letting the tmt test generate results.yaml could resolve this? then we can run rpminspect as the user would, the only magic would be at the parser of results which would translate it to tmt's results.yaml?

For sure. Or simply generating the report at the end and pointing people directly to the report in the workdir.

@msrb the html generation is also a solution, but if it is results.yml, the report will always look the same.

I have filled:

https://github.com/teemtee/tmt/issues/1074

Feel free to extend it with your ideas