Describe what you would like us to do:

Allow the DistroBuildSync tool to send messages on fedora_messaging bus using the topic org.fedoraproject.{prod,stg}.buildsync.eln.enqueue, as well as allowing the distrobuildsync-eln queue to receive it.

Reasoning: The sync tool processes builds in batches, triggered by org.fedoraproject.prod.buildsys.tag messages. Sometimes, the sync needs to process one or more of these ahead of the others, so we want to push the remainder back onto the queue of incoming messages so they don't get lost in the event of a crash. We could do this by resending org.fedoraproject.prod.buildsys.tag messages, but that has the potential of confusing any other consumer listening for that event.

When do you need this to be done by? (YYYY/MM/DD)

As soon as possible, to avoid further issues around packages such as llvm and ocaml.

zlopez commented a year ago

From what I remember we need a new cert for this and probably updating the org.fedoraproject.*.buildsys schema if there is any.

Whoever will take the issue it would be nice to create a guide, how to add a new message topic to fedora messaging. I couldn't find any.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-trouble, medium-gain, rabbitmq

a year ago

sgallagh commented a year ago

To be clear, this was a request for org.fedoraproject.*.buildsync, not org.fedoraproject.*.buildsys. If that might be confusing, I'm open to other suggestions. Maybe rebuilder instead of buildsync?

Upon further reflection, I was also thinking that we should request a couple other topics while doing this. Could we add:

org.fedoraproject.*.rebuilder.eln.enqueue
org.fedoraproject.*.rebuilder.eln.batch.create,
org.fedoraproject.*.rebuilder.eln.batch.state_change,
org.fedoraproject.*.rebuilder.eln.attempt.create
org.fedoraproject.*.rebuilder.eln.attempt.state_change

This would allow us to set up notifications for batch rebuild state as well as retry attempts.

zlopez commented a year ago

I actually missed the buildsync. In this case this will need to be a new fedora messaging schema.

sgallagh commented a year ago

I actually missed the buildsync. In this case this will need to be a new fedora messaging schema.

What can we expect for a timeframe?

zlopez commented a year ago

@abompard I think setting up the new queue in rabbitmq shouldn't take much. Am I right?
@sgallagh You should provide the schema if this is a new publisher, but we already have few apps that are just publishing messages without schema, so it's not a hard requirement. See https://fedora-messaging.readthedocs.io/en/stable/user-guide/messages.html

Edited a year ago by zlopez

abompard commented a year ago

PR #1610 should:
- allow the distrobuildsync-eln user to publish to any topic below the org.fedoraproject.{stg,prod}.rebuilder.eln. prefix
- subscribe the distrobuildsync-eln queue to the new topics you listed.

Is this what you wanted @sgallagh ?

sgallagh commented a year ago

PR #1610 should:
- allow the distrobuildsync-eln user to publish to any topic below the org.fedoraproject.{stg,prod}.rebuilder.eln. prefix
- subscribe the distrobuildsync-eln queue to the new topics you listed.

Is this what you wanted @sgallagh ?

Almost exactly! Though that revealed why we had a different bug that I had to work around in the ELNBuildSync code. We weren't receiving some notifications, but apparently that's because we needed to tell you which ones to send us. (I thought our client config handled that with its routing_keys setting).

sgallagh commented a year ago

I'll work on adding a message schema. I assume I follow the instructions at https://fedora-messaging.readthedocs.io/en/stable/tutorial/schemas.html and then send a merge request to the fedora-messaging repo?

abompard commented a year ago

Nope, the message schemas are a separate python package that you own, publish and update when you deem it necessary. Consumers of the messages will only install this package. Feel free to ask if you need any assistance with this, and/or if the docs aren't clear.

sgallagh commented a year ago

Nope, the message schemas are a separate python package that you own, publish and update when you deem it necessary. Consumers of the messages will only install this package. Feel free to ask if you need any assistance with this, and/or if the docs aren't clear.

I used koji-fedoramessaging-messages as inspiration and put together https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/elnbuildsync-messaging which I would welcome a review of, if you have a chance.