Note: if the content of the message as it is now does not satisfy you, we are more than happy to accommodate :)

@pingou the change should be possible, do you already have an example of the message content?

do you already have an example of the message content?

Not yet as they are not live yet. The current schema is defined at: https://github.com/fedora-infra/bodhi/blob/develop/bodhi/messages/schemas/update.py#L211

As said in https://pagure.io/fedora-ci/general/issue/70#comment-596192 we're more than happy to accommodate you if you need more information in the message

Also, we've https://github.com/fedora-infra/bodhi/issues/3497 which will land in the schema when implemented which will likely be relevant for you

do you already have an example of the message content?

Not yet as they are not live yet. The current schema is defined at: https://github.com/fedora-infra/bodhi/blob/develop/bodhi/messages/schemas/update.py#L211
As said in https://pagure.io/fedora-ci/general/issue/70#comment-596192 we're more than happy to accommodate you if you need more information in the message
Also, we've https://github.com/fedora-infra/bodhi/issues/3497 which will land in the schema when implemented which will likely be relevant for you

Would be possible for the message to contain the taskID of the build and also the Fedora release?

Would be possible for the message to contain the taskID and also the Fedora release?

The taskID I doubt so, bodhi doesn't record this (note: you can use bodhi updates download to download the builds of an update, which I suspect is what you had in mind?)

The Fedora release sounds doable, I am actually surprised it's not in there already.
I've made a ticket to track this: https://github.com/fedora-infra/bodhi/issues/3515

Would be possible for the message to contain the taskID and also the Fedora release?

The taskID I doubt so, bodhi doesn't record this (note: you can use bodhi updates download to download the builds of an update, which I suspect is what you had in mind?)

The problem is the CI message we publish requires the taskID, issuer and source. Apart from other information, but these 3 items I'm not sure how to get them if they are not in the trigger message.

https://pagure.io/fedora-ci/messages/blob/master/f/schemas/rpm-build.yaml

The Fedora release sounds doable, I am actually surprised it's not in there already.
I've made a ticket to track this: https://github.com/fedora-infra/bodhi/issues/3515

But you're not testing a build here you're testing an update, so the rpm-build message is likely to indeed not fit.

The Fedora CI pipeline tests builds, so for bodhi trigger my idea was we we run the tests on Fedora CI for each build listed on this bodhi updates. This is how we do downstream with build group. For this to work we would need to know details for each of the builds.

@mvadkert what do you think?

@bgoncalv I think it should be possible to query Koji for details about builds. CI systems could get all the information (task ID, issuer, etc.) they need that way.

@bgoncalv I think it should be possible to query Koji for details about builds. CI systems could get all the information (task ID, issuer, etc.) they need that way.

Right, it is possible, but the way the pipeline works today we check if the component for the branch has tests, if not we don't even schedule the test run. This is done on jenkins master where we don't have koji installed and I don't know a good way to query this info without using koji.

@bgoncalv ah, I completely understand now. Nobody wants to touch XML-RPC in 2019 :P I am planning to create a small (incomplete) wrapper around Brew XML-RPC so people can use it in Jenkins, as a shared library.

In the mean time, following command would give you the required payload for your XML-RPC call:

$ koji -d --debug-xmlrpc search build --exact xmlrpc-3.1.3-20.fc28

That would give you a build ID. Another call would be required to get task ID for the build.

@pingou is is unrealistic to use koji-build-group.build.* messages ?

https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.build.complete.json

That is what we use on downstream for sidetag testing

@msrb

@bgoncalv ah, I completely understand now. Nobody wants to touch XML-RPC in 2019 :P I am planning to create a small (incomplete) wrapper around Brew XML-RPC so people can use it in Jenkins, as a shared library.

Did you know that already this should already exist, continous productization should have it?

@mvadkert really? Last time I check they were calling the brew CLI under the hood. But maybe they have more implementations somewhere (?)

aha, so could be :)

@pingou is is unrealistic to use koji-build-group.build.* messages ?
https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.build.complete.json

Sorry for taking so long to get back to you on this.
Do you know which of these fields would be the required ones? All of them?
The source and id fields in the list of builds, for example, would mean that bodhi queries koji for each build in the update and when the update contains 100s of builds this may become quite slow.
Type, issuer, component, nvr, scratch on the other hand should be no problem.

Do you know what the repository field in artifact is supposed to point to?

Finally, if the payload is the same (or similar) but the topic different, is that an issue for you?

@pingou is is unrealistic to use koji-build-group.build.* messages ?
https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.build.complete.json

Sorry for taking so long to get back to you on this.

Np :)

Do you know which of these fields would be the required ones? All of them?

The schema is here with these as required fields:
https://pagure.io/fedora-ci/messages/blob/master/f/schemas/rpm-build.yaml#_78

The source and id fields in the list of builds, for example, would mean that bodhi queries koji for each build in the update and when the update contains 100s of builds this may become quite slow.

Hmm, currently id is definitely required for all artifacts :( as it is the unique for all brew builds including scratch :( I am not sure we could workaround it ... maybe we could start using as ID the NVR which would also imply that the build is not scratch ... but yeah .. not happy about it, but I get the point.

@psss @happz @ralph what do you think guys?

Type, issuer, component, nvr, scratch on the other hand should be no problem.
Do you know what the repository field in artifact is supposed to point to?

Repository we use to provide ODCS repository to the sidetag. I think it should be optional (it is required now) ....

https://pagure.io/fedora-ci/messages/blob/master/f/schemas/rpm-build-group.yaml

Finally, if the payload is the same (or similar) but the topic different, is that an issue for you?

I would at least go with suffix in topic if not a problem:
i.e. your-topic.koji-build-group.build.complete

This message will trigger the Fedora CI pipeline, the pipeline relies on koji task id as it also runs for scratch builds, to avoid big change in the pipeline or have the pipeline to query koji for the task id it would be nice if the message would contain this id of the builds.

For the ID does it need to be the koji task id? Could it be the bodhi update id (versioned)?

repository in the case of bodhi could be a link to the update in bodhi?

Using the same suffix should be fine I think.

Arf, I've just read @bgoncalv 's reply which answers my first question :(

Looking at the code with @cverna it seems we may be able to retrieve that build id \ó/

We'll be looking into this this week.

New format implemented in: https://github.com/fedora-infra/bodhi/pull/3629

nice, it seems to contain all information we need :)

Messages are now being sent in staging: https://apps.stg.fedoraproject.org/datagrepper/id?id=2019-02b72ce1-69e1-415b-81d9-88187a584595&is_raw=true&size=extra-large

cool, it would be nice if the message could contain the fedora release like F30 or Fedora30 and for rawhide rawhide.
Could be for example part of the id: https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.build.complete.json#_12

or some new field...

I went with the new field in: https://github.com/fedora-infra/bodhi/pull/3672 let me know if this isn't enough

It is much better, unfortunately it will not return rawhide, for rawhide builds.
It would be much better if this could be done on the message side, as ew don't have bodhi-clinet installed on our side it can be tricky for use to know what is the current fedora relased used by rawhide.

It is not ideal for Bodhi to send rawhide since there is no concept of a rawhide Release in Bodhi.
A release in Bodhi has 3 possible state, Archived, Current and Pending. Rawhide is the latest pending release.

You can actually query these without using the bodhi client, using the following endpoint https://bodhi.fedoraproject.org/releases/?state=pending

Thanks, that will help us.

New bodhi is out and the message format has been adjusted to include the release info: https://apps.stg.fedoraproject.org/datagrepper/id?id=2019-b5a02110-ce90-4237-943f-f023fd2e85a2&is_raw=true&size=extra-large

Let us know if there is anything else we can do :)

nice! glad to see standardized messages floating !

I've checked https://apps.fedoraproject.org/datagrepper/id?id=2019-9dbe37a6-7fd8-4fe4-8f15-bd2abee7919e&is_raw=true&size=extra-large

the id in a build is the buildID, we need to run the tests the taskID

hub.getBuild(1412775)
{'cg_id': None, 'package_name': 'perl-HTTP-Date', 'extra': {'source': {'original_url': 'git+https://src.fedoraproject.org/rpms/perl-HTTP-Date.git#26fca451772753a8444b6bb3cb45f7dc56249eeb'}}, 'creation_time': '2019-11-14 07:36:34.963623', 'completion_time': '2019-11-14 07:37:34.464488', 'package_id': 11747, 'cg_name': None, 'id': 1412775, 'build_id': 1412775, 'epoch': None, 'source': 'git+https://src.fedoraproject.org/rpms/perl-HTTP-Date.git#26fca451772753a8444b6bb3cb45f7dc56249eeb', 'state': 1, 'version': '6.03', 'completion_ts': 1573717054.46449, 'owner_id': 1374, 'owner_name': 'ppisar', 'nvr': 'perl-HTTP-Date-6.03-1.fc32', 'start_time': '2019-11-14 07:36:34.963623', 'creation_event_id': 48709800, 'start_ts': 1573716994.96362, 'volume_id': 0, 'creation_ts': 1573716994.96362, 'name': 'perl-HTTP-Date', 'task_id': 38987248, 'volume_name': 'DEFAULT', 'release': '1.fc32'}

from koji getBuild method it is on task_id field.

We also need the source more specifically the field extra['source']['original_url']

Note that the source field is not marked as mandatory in: https://pagure.io/fedora-ci/messages/blob/master/f/schemas/rpm-build.yaml

Which in the case of bodhi is a good thing since accessing it would be at least one more call to koji for each build in an update :(

@pingou okay by me if the standard doesn't require source I'm fine by not setting it in the CI message.

yeah, it is optional ...

The new messages are being published in staging: https://apps.stg.fedoraproject.org/datagrepper/id?id=2019-376d2ad3-5c2b-461d-8693-db439b630a91&is_raw=true&size=extra-large

Fedora CI now triggers on bodhi updates messages.

So, this might be nice for what CI expects, but the way the message is nothing at all like the other messages is kind of weird and not at all what the openQA scheduler expects.

I want to implement scheduling on these messages in openQA so we will schedule tests on updates which fail their CI 'push to testing' gating requirements (right now we use the 'push to testing request' messages as a proxy for 'update was created', but since updates can now be gated from ever being submitted to testing, that doesn't entirely work any more), but the way this message's format is nothing like any of the other messages we consume makes that unnecessarily difficult. So instead of making the openQA scheduler a lot messier, I sent a PR to make this message work for everyone (I hope):

https://github.com/fedora-infra/bodhi/pull/4381

Sorry to necro this again, but I'm looking at this area again (test triggering) in the context of https://pagure.io/fedora-ci/general/issue/436 (trying to trigger tests earlier in Fedora CI).

This caused me to look in more detail at how much of the stuff that Bodhi put in the messages in response to this ticket is actually used by CI, and my conclusion is "very little of it".

Can folks look over https://pagure.io/fedora-ci/general/issue/436#comment-872382 and the subsequent comment and see if I'm missing anything? I would like to rejig things to make scheduling much simpler and more consistent across CI and openQA, by just having the update dict that's present in every message (including the current version of the koji-build-group.build.complete message) contain the additional information CI needs.