Hi, I work on the Podman Desktop GUI for podman, and it would be very nice to have automated testing for Podman Desktop in Fedora OpenQA, particularly if Podman Desktop tests could be triggered by updates to podman, which is already in Fedora OpenQA.
There are a few complications to getting it into OpenQA:
Would it be possible to investigate what it would take to pull a flatpak from flathub into Fedora OpenQA to enable testing of Podman Desktop?
OK, so, this is a bit different from my understanding on IRC, but - good news! - in a good way.
Having openQA install something from flathub is totally trivial. That's not a problem at all. The more difficult bit is if we want to run tests when something in Podman Desktop changes and communicate the results to Podman Desktop upstream, because all the scheduling/reporting bits aren't really...set up for that.
But if the request is just "test Podman Desktop when podman changes", that is actually easier. We already run tests when podman changes. All we need to do is add another test which installs Podman Desktop and does whatever stuff you want to do in it. That's not hard at all, we just have the test run a flatpak install, simples. Of course, the test would need to be kept up to date for changes in Podman Desktop itself, but that's not so different from various other tests we have.
flatpak install
So...we can definitely do this much. Can you specify what exactly you would like us to test in Podman Desktop? Ideally in a way that's understandable for someone who's never run it? :D
Hey @adamwill, If it is possible the aim would be something like on github actions: https://github.com/containers/podman-desktop/blob/main/.github/workflows/pr-check.yaml#L218.
Requires nodejs 18, yarn installed, xvnc or xvfb so we can run e2e tests. I am not sure what is your environment and what automation could do there.
So, running this kind of test suite within openQA feels like a bit of an "impedance mismatch" to me generally speaking (and it can be tricky to run it on the app when installed "conventionally", sometimes they assume they are running from within a copy of the code repo, though I don't know if that's the case here).
What I would likely do instead is just reimplement that in 'native' openQA test code - that is, write an openQA test that just runs podman-desktop and follows basically the same steps (check it sees the startup screen, click through the analytics prompt, check it reaches the main screen), using openQA's native approach (screenshot matching).
We do something similar with Cockpit, where Cockpit upstream has a very extensive set of selenium-based tests, and openQA tests a smaller subset of the same functionality, but using screenshot matching to test as an end user would see it; this turns out to be quite valuable, because openQA can catch stuff that just introspecting the DOM can't, like a font looking wrong or missing characters, for instance.
I do see that the test framework podman-desktop uses has some basic screenshot matching functionality, which is cool, but openQA's may be more sophisticated (it matches areas, does some configurable fuzzing, has a 'tag' system for flexibility with letting different images match when appropriate, and can click on the stuff it matches).
if it's possible to run the upstream test very easily on podman-desktop installed to a regular Fedora system via flatpak, we could look at doing it, though. I just don't know if that's the case.
Metadata Update from @adamwill: - Issue tagged with: newtest
I am not familiar with openQA at all so it is hard to precisely decide what way could be better, nor I can tell if rewriting would make things easier...
To run the upstream e2e tests you need to build the project using node and yarn. Then some virtual buffer is needed to work with UI of the app, upload the result in some way, report it.
We could avoid all this if we can install app using flatpak file (for example available from github repo on nightly builds or from a stable release tag) and then run these openQA based tests. They just need to interact with the electron app through UI in order to perform some scenario.
Do you happen to have such example somewhere?
Right, that's pretty much what I figured: it'll be easier to just write an openQA "native" test that exercises the basic UI features, than try to execute the upstream test in an installed system environment.
An example would be the Cockpit tests - e.g. https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/tests/server_cockpit_basic.pm . It's not exactly rocket science, just "look for this, click it, type this" and some logic for handling cases where a click gets lost or something. The 'needles' live under https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/needles/cockpit , they are basically a screenshot paired with a JSON metadata file defining specific areas of the screen to be matched and some other stuff.
Anyway, we would write the test, you don't have to. It would be a part of this project, which is the Fedora openQA test suite, which we (Fedora QA) maintain and are responsible for. It seems like a reasonable thing to add to that test suite which would provide value for both Fedora and upstream so I've no problem with doing it, just a case of finding the time (or @lruzicka might do it also). If you're super keen by all means PRs are welcome, but they're not necessary and it is a bit of a trek to get an openQA instance up and running and learn to write tests. https://fedoraproject.org/wiki/OpenQA is the starting point if you really want to try, though.
@adamwill That sounds good! I would be interested, but has zero capacity, if you folks can (and I believe you are gonna be much faster) do it, it would be awesome. I can provide a PR review or any other data for proper test to be written.
@odockal, I am not very much experienced with Podman, but I can help you to write the openQA part if you give me a procedure to test what you need to be tested. Thanks.
yeah, sorry, I did not get back to this yet :( it's on my large teetering pile of "stuff it'd be cool to do but which isn't actually on fire".
@lruzicka if you want to take a swing at this, I think the place to start is the .ts files in https://github.com/containers/podman-desktop/tree/main/tests/src . Similar to openQA tests, you can sort of see the essentials of what they're testing even if you don't understand the test framework - the comments and text for the actual test steps are handy. The idea would be, I think, to reimplement some or all of those tests as openQA tests.
Log in to comment on this ticket.