#36 Move Taskotron tests into Fedora CI
Opened 5 years ago by kparal. Modified 3 years ago

This is a tracker ticket for moving all existing Taskotron tests into Fedora CI. The goal is to have a single CI system to support and maintain, instead of two.

I'll link here tickets for issues we're blocked on and keep this ticket description up-to-date.

Basic info

Currently, we have the following tasks in Taskotron active:

Everything except rpmdeplint is scheduled immediately after a new Koji build is built and tests that build. Rpmdeplint is scheduled each time a Koji tag (e.g. f29-updates(-testing)-pending) is updated (a new build is tagged into it) and tests all builds present in that tag every time. Due to its dependency checking goals, there are good reasons for this behavior. However, because it works completely differently than it is common for other tests, it brings many new challenges for the test system. Either we can see this as a good thing to make the test system more generic and future-proof, or we can consider re-working rpmdeplint behavior from the ground up to make it easier for the current test system - just be aware that that problem area is full of tricky issues and corner cases and it's definitely not simple.

Tickets we're blocked on

Optional tickets

Meeting logs links


I've filed tickets for all the requirements I've raised during 2019-03-06 meeting.

Can we specify some quality metrics that should be met by Fedora CI before me move something that's working, usable and reliable into something that's not (yet?)?

If there were some issues with executing in Fedora CI, we could of course keep running it in Taskotron until the issues are resolved (it might take a long time until the necessary features mentioned above are implemented anyway). But since Fedora CI is likely to get used for gating in mid-term future, I presume that there will be a strong push for having its reliability very high. Fwiw, Taskotron reliability is pretty decent nowadays, I believe, but it's nowhere perfect, and most importantly people can't reschedule tests when needed. I hope that this important feature will arrive in Fedora CI instead, especially when it's used for gating.

I've tried to update the tickets with short summaries of our today's in-person discussion with @mvadkert @psss and @jskladan.

Metadata Update from @bookwar:
- Issue tagged with: new pipeline

5 years ago

Per https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/TRQZCXI5IZH3GLJTCO56SMBX5UY6J6LL/, taskotron went EOL as of last Friday.

From the linked issue:

--------------------------------
Is anything replacing Taskotron?
--------------------------------

In the short term, there is a Jenkins instance which is running
rpminspect. Rpminspect includes tests which are effectively the same as
rpmlint and rpmgrill which removes the need to run them.

We already have rpminspect results going into production resultsdb
and showing up in Bodhi.

In the longer term, there will be a new solution to run rpminspect,
rpmdeplint and other static-analysis style checks coming from the Fedora
CI folks which they will be deploying before Flock.

As this reads, I believe this task can be closed; if not, then we should identify and list tasks that need to be completed.

As this reads, I believe this task can be closed

I don't agree. I remember there were a least 3 useful checks available:

  • rpmdeplint
  • rpmlint
  • abi breakage (I forgot the name)

None of them is available now, rpminspect is not fully covering for rpmlint.

@churchyard I'll work to compare the state of today's test vs what we had before, and what we need to close the gap. I assumed incorrectly that the service going EOL signified everything had in fact been migrated. :(

No in fact, Taskotron went EOL before there was feature parity, which makes me incredibly sad :(

As this reads, I believe this task can be closed

I don't agree. I remember there were a least 3 useful checks available:

rpmdeplint
rpmlint
abi breakage (I forgot the name)

None of them is available now, rpminspect is not fully covering for rpmlint.

As far as I knew, the Fedora CI folks have been working on a Jenkins instance to run rpmdeplint in Fedora and it seemed like a waste of effort to port the Taskotron task. Did I misunderstand something?

I was under the impression that rpminspect covered everything that rpmlint does - is that not the case?

The one which may have fallen through the cracks is abicheck. I remember having a conversation around whether we were going to port that one and I remember coming to the conclusion that it wasn't needed but I honestly don't remember how we came to that conclusion or who else was involved in the conversation.

I was under the impression that rpminspect covered everything that rpmlint does - is that not the case?

I was under the impression that rpminspect does different (possibly more relevant) checks that somewhat overlap with what rpmlint does. We have some rpmlint configuration for our Python packages and we have used rpmlint to verify nothing changed in bad direction. I wonder where to see the actual results of rpminspect and how to migrate our config.

As far as I knew, the Fedora CI folks have been working on a Jenkins instance to run rpmdeplint in Fedora and it seemed like a waste of effort to port the Taskotron task. Did I misunderstand something?

Nope, you're right. It is coming -- the plan is to have it running by the end of this month.

The one which may have fallen through the cracks is abicheck. I remember having a conversation around whether we were going to port that one and I remember coming to the conclusion that it wasn't needed but I honestly don't remember how we came to that conclusion or who else was involved in the conversation.

Note once we have all the pieces (generic pipelines + library, Testing Farm, Jenkins instance) up and running, it should be fairly easy to add new generic tests. That is one of the goals. I don't know much about abicheck so I don't know if it would require any special treatment, but if people agree that it would be beneficial to have it, then we could use it as a test of we are designing and building now.

Abicheck is very useful, yes, and it would be beneficial to have it executed. The point of contact is @dodji.
I don't know whether @churchyard wants to have his python-versions running in Fedora CI.
The rest of tasks is covered, with caveats (rpminspect is not completely the same as rpmlint, but it might be a superset, I didn't study it - @dcantrell should know).

Please understand that all this was not a hard requirement, this was a plan for smooth transition (which was delayed several times). It was also an attempt to specify all missing features which we needed. The ship has sailed, Taskotron is now EOL, so this ticket is no longer valid in the same form it was written. Some tests were implemented differently and not taken from our Taskotron setup (i.e. rpmdeplint running on a custom pipeline), some got obsolete (rpmgrill). The linked implementation tickets are still valid, I believe, as general ideas of what features would be good to implement and which use cases to bear in mind. Don't feel strictly tied to them. Do whatever you feel is appropriate to do with these tickets. They can be closed and some specific ones (e.g. for abicheck) can be opened, if you feel that's better. The overarching main request is still valid and simple - different teams in Fedora (not just Fedora QA) might be interested to implement a generic cross-distribution check and Fedora CI should offer a reasonable way to do so.

FWIW I still prefer to run rpmlint together with rpminspect. I don't think one is superset of the other.

About the Python checks, no, we don't consider them necessary any more. We are notified differently, when people go crazy. They were useful where there was plenty Python 2 packages, today, not so much.

@churchyard is correct regarding rpmlint vs. rpminspect. rpminspect does not completely replace rpmlint, though it may be able to do that at some point in the future. I have work pending to get the output style @churchyard requested from rpminspect (I haven't forgotten!).

Yesterday I added the dangling symlink inspection to rpminspect. It also guards against builds that turn directories in to symlinks which is something rpm can't handle on upgrades.

Please have a look at the TODO file in rpminspect and let me know which of the outstanding inspections should get higher priority for Fedora CI. @dodji is looking at libabigail integration. I completed the symlinks check and I will continue moving on to outstanding work items in my existing priority order. Do open RFEs for rpminspect in github to help me prioritize.

Thanks.

It also guards against builds that turn directories in to symlinks which is something rpm can't handle on upgrades.

That is a perfect feature!

Login to comment on this ticket.

Metadata