#245 [WIP] Add openQA tests for i3
Opened 2 years ago by defolos. Modified 8 months ago
fedora-qa/ defolos/os-autoinst-distri-fedora i3-tests  into  master

No commits found

This is a huge PR adding the i3 spin to the Fedora openQA tests. So far the i3 spin is only added to a few existing test suites where it appeared appropriate and to the new test suite i3-tilling.

I have gotten the test suite locally on my machine by applying the fif templates and scheduling the i3 product via:

$ openqa-cli api -X POST isos ISO=Fedora-i3-Live-x86_64-Rawhide-20210721.n.0.iso DISTRI=fedora VERSION=Rawhide FLAVOR=i3-live-iso ARCH=x86_64 BUILD=Rawhide-20210721.n.0 RAWREL=35 DEPLOY_UPLOAD_TEST=install_default_upload

where I put the latest Rawhide compose i3 iso into /var/lib/openqa/factory/iso.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

I assume you don't want this considered for merging till you remove the [WIP], right? Thanks!

Well, it works on my system, but it's not properly wired into fedora-openqa=
and it probably doesn't meet your requirements, that's why I marked it as =
WIP (I'd be more than happy to get this merged sooner rather than later)=2E=
So by all means please consider it for merging=2E

Perhaps this second time is not needed any more?

I also wonder, what tool are you using to create the needles? This is the first time I have seen click_point defined in them. When OpenQA calculates the click point, it basically halves the width and height values, so basically it always arrives at that exact point you are providing.

So, I tried to run the suite in my OpenQA, but on my computer, there are some problems that cause failures. My observations are based on the i3-tests branch of defolos's fork:

  • in templates.fif.json the product is missing the DEPLOY_UPLOAD_TEST which causes the tests run ad hoc without waiting for the install_default_upload to finish and upload the assets.
  • Tests desktop_browser, desktop_login, desktop_printing, desktop_postinstall_notifications and i3_tilling are failing on non-matching needles. The needles are there, but they only partially match (40-70%) - I have noticed that you tend to compare multiple areas - in my experience, this leads to failing needles because the chance of hitting a difference on a needle multiplies. We do not use many such needles in our current setup.

You say that the suite passes on your system, which ISO have you used to create the needles?

rebased onto e5c70c0ccf55a4b1f96b88e65fd34ca4ed7b9908

2 years ago

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

rebased onto 5620054addd32a31016037173c3a7feb4b52e482

2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

Perhaps this second time is not needed any more?

Whoops, that was not intentional. I've removed it now.

I also wonder, what tool are you using to create the needles? This is the first time I have seen click_point defined in them. When OpenQA calculates the click point, it basically halves the width and height values, so basically it always arrives at that exact point you are providing.

I have mostly used the webui, but I've also modified a few needles using the openQA needle editor.

So, I tried to run the suite in my OpenQA, but on my computer, there are some problems that cause failures. My observations are based on the i3-tests branch of defolos's fork:

  • in templates.fif.json the product is missing the DEPLOY_UPLOAD_TEST which causes the tests run ad hoc without waiting for the install_default_upload to finish and upload the assets.

Ah, I see. I forgot to add this variable to the fedora-i3-live-x86_64 product. That should be fixed now. I worked around this by setting the variable DEPLOY_UPLOAD_TEST in the opeqa-cli invocation.

  • Tests desktop_browser, desktop_login, desktop_printing, desktop_postinstall_notifications and i3_tilling are failing on non-matching needles. The needles are there, but they only partially match (40-70%) - I have noticed that you tend to compare multiple areas - in my experience, this leads to failing needles because the chance of hitting a difference on a needle multiplies. We do not use many such needles in our current setup.

You say that the suite passes on your system, which ISO have you used to create the needles?

I have used the latest Fedora Rawhide Compose (when submitting, that was build 20210721.n.0). I'll make a run with 20210812.n.1 overnight and will check if it still passes locally or fix any errors that pop up.

rebased onto a76cfc3cefb34e1b340b1ff6f499a9500b95c25a

2 years ago

Build succeeded.

rebased onto 693afdf58df061967e2fb4dbbaad6669dfa124c2

2 years ago

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

rebased onto d9e6d89

2 years ago

Build succeeded.

@lruzicka I've been running the i3 openqa tests for the last few composes successfully on my local box.

@lruzicka I've been running the i3 openqa tests for the last few composes successfully on my local box.

Sorry, did not see this. I am going to try it now.

@lruzicka @defolos we should really get back to this, but the pull request will need a lot of rebasing at this point, I guess.

Metadata