They don't do the same thing in technical terms; the openQA result publishing doesn't use the CI messages at all (primarily because they didn't exist when we wrote the result publishing).

But they do the "same thing" in terms of what is ultimately achieved: publish a result to resultsdb for completed tests.

It's been long enough since we wrote this stuff and the things that use it that I don't remember all the ins and outs off the top of my head, and I'm not familiar with exactly how ci-resultsdb-listener works or what format it publishes results in. We'll have to look into the details. The most important thing that consumes the results is Greenwave; there are also fedora_nightlies and check-compose, off the top of my head.

The existing reporting code is here. It uses resultsdb_conventions, which was my attempt to write something to make it convenient to report results in consistent format from multiple test systems, but no other systems really adopted it.

The current CI message publishing code is here. You can obviously see how typical results and CI messages look in the resultsdb web UI and datagrepper.

I was actually wondering if the logs/message format could be configured from an environment variable or even just loading a python file from a config file to deal with it, this way we could have one flexible app - by one app I mean one code base.

honestly I'd kinda rather keep things as consistent as possible :D bodhi/greenwave already have to do some dumb stuff to account for different resultsdb formats. I think basing the resultsdb message on the - fairly well standardized and documented/schema'd - CI format would be great overall. just need to find time to dig into the details and make adjustments to things that consume the results where appropriate. right now there's all this fire all over the place...

sorry I haven't got to this yet :( it's constantly the next thing on my list at the end of the day, then something sets on fire the next morning...

So, now F35 is done, I have some time for this. One thing that immediately comes up is: what's the relationship between ci-resultsdb-listener and resultsdb-updater? They appear to do much the same thing. @mvadkert @lholecek

For using ci-resultsdb-listener, I see several issues right away. It basically assumes all messages will be for Koji builds, it makes several assumptions of that. In the building of the data dict:

    data = {
        "build_url": message.body["run"]["url"],
        "repo": message.body["artifact"]["component"],
        "nvr": message.body["artifact"]["nvr"],
        "item": message.body["artifact"]["nvr"],
        "scratch": message.body["artifact"]["scratch"],
        "username": message.body["artifact"]["issuer"],
        "koji_task_id": message.body["artifact"]["id"],
        "namespace": message.body["test"]["namespace"],
        "type": message.body["artifact"]["type"].replace("-", "_"),
        "test_type": message.body["test"]["type"],
        "category": message.body["test"]["category"],
    }

lots of those are unsafe assumptions, particularly the things pulled from the "artifact" dict. If you look at the message schemas, in the koji-build schemas - e.g. koji-build.test.complete - the artifact is defined as an instance of the rpm-build schema, and that has all the properties ci-resultsdb-listener expects. But that's not the only thing an 'artifact' can be! If you look at the fedora-update schemas - e.g. fedora-update.test.complete - the artifact in those messages is defined as an instance of the fedora-update schema. And that doesn't have the same properties as an rpm-build at all, and this code would choke on it all over the place. Ditto productmd-compose messages; for these the artifact is a productmd-compose, which again doesn't have the same properties at all.

Just from this quick look, the approach of resultsdb-updater seems superior, as it doesn't keep this kind of mapping logic 'in-house' but uses mappings that are stored in the message schema repository; so I could theoretically just write mappings for the topics openQA publishes on, and the tool would then be able to report the results.

@kevin just tagging you for info - this is one way to fix the "openQA can't report to resultsdb if auth is enabled" problem. The other way would just be to add auth support to fedora-openqa's own resultsdb reporting code, but I do like the idea of using one of these systems if we can make it work.

@adamwill

One thing that immediately comes up is: what's the relationship between ci-resultsdb-listener and resultsdb-updater

I guess no relationship, this piece (resultsdb updater from amqp messages) was never unified.

resultsdb-updater is being deprecated on downstream in favor of Hydra.

@lholecek @draval @timitra is it unreal to expect Hydra could be sometime deployed in Fedora for fedora-mesagging? So we finally get rid of this unneeded duplicity?

@mvadkert @lholecek @draval @timitra ping? Where does this hydra thing live, anyway? Is it inside the firewall?

AFAIK, it's an internal redhat project that lives inside the firewall, yes

On Mon, Jan 24, 2022 at 2:06 PM Adam Williamson pagure@pagure.io wrote:

adamwill added a new comment to an issue you are following:
@mvadkert @lholecek @draval @timitra ping? Where does this hydra thing live, anyway? Is it inside the firewall?

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-qa/fedora_openqa/issue/90

Yeah, I'm following up on that internally and have found it now. It's currently RH internal-only and proprietary. I'm trying to see if this can be changed. If it stays that way, we can't really use it in Fedora, and would I guess have to keep shadowing it with ci-resultsdb-listener (and contribute improvements there) or adopt the 'obsolete' resultsdb-updater.

I believe we are moving in a direction we will submit things directly to ResultsDB. Right @msrb ?

oh, well that seems sort of a shame? I liked the concept of CI forwarding, because we have schemas for CI messages, so if we forward CI results to resultsdb we get a lot of consistency 'for free'. Whereas we don't have a successful schema-type thing for resultsdb (since nobody else adopted my resultsdb_conventions project or proposed an alternative), so there's nothing imposing any kind of consistency on resultsdb data. There is a lot of awkward inconsistency between existing resultsdb results which causes some awkwardness in greenwave and bodhi, and if more things start reporting direct to resultsdb with no kind of framework for consistency this might just get worse...

Just a bump on this - I did talk to @mvadkert about it (I think it was him, anyway) again somewhat recently, and the "just have everyone report to resultsdb directly" path may actually be the most sensible one, thinking it through. The thing is, resultsdb publishes messages when you submit a result to it, so looked at from a certain angle, publishing a message then submitting a result to resultsdb is kinda unnecessary duplication (we get two messages for every event, effectively, especially now we're submitting "results" for queued/running tests).

so, ultimately it might make sense to go down a path where openQA and Fedora CI (and any other system that gets invented, there still may be new ones, I'm talking to the CoreOS team about that recently, they may want to 'onboard' their own CI system so that it test updates and get results shown in Bodhi etc.) don't bother publishing ci. messages, they just send results to ResultsDB - ideally in a format that's shared and similar to the ci-messages format - and anything that wants to consume a message just consumes the resultsdb.result.new message. That would reduce complexity quite a bit.

I was hoping to do something at Flock to talk about this, but not sure enough relevant folks will be there, I don't think Miro will be. I'll talk to the CoreOS folks about it some more though, at least.

yeah, I will be not there, but @siwalter from Testing Farm will be, but he might not have all the details :(

Anyway, I talked to CoreOS folks and seems they were interested in using Testing Farm (what is the backend Fedora CI), so I would like to learn more how can we help them to rather provide them Fedora CI for the stuff they need, instead of they rolling their own CI system.

Any contact points for that discussion? I personally talked to Timothée Ravier there

Yeah, that's who I talked to at devconf, I believe also @dustymabe and @jlebon were interested (I sent them a long email with tech details back in May which was about this topic, I think).