#11585 New AMQP message: org.fedoraproject.prod.buildsync.eln.enqueue
Opened 6 months ago by sgallagh. Modified 3 months ago

Describe what you would like us to do:


Allow the DistroBuildSync tool to send messages on fedora_messaging bus using the topic org.fedoraproject.{prod,stg}.buildsync.eln.enqueue, as well as allowing the distrobuildsync-eln queue to receive it.

Reasoning: The sync tool processes builds in batches, triggered by org.fedoraproject.prod.buildsys.tag messages. Sometimes, the sync needs to process one or more of these ahead of the others, so we want to push the remainder back onto the queue of incoming messages so they don't get lost in the event of a crash. We could do this by resending org.fedoraproject.prod.buildsys.tag messages, but that has the potential of confusing any other consumer listening for that event.

Related: https://github.com/fedora-eln/eln/issues/169

When do you need this to be done by? (YYYY/MM/DD)


As soon as possible, to avoid further issues around packages such as llvm and ocaml.


From what I remember we need a new cert for this and probably updating the org.fedoraproject.*.buildsys schema if there is any.

Whoever will take the issue it would be nice to create a guide, how to add a new message topic to fedora messaging. I couldn't find any.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-trouble, medium-gain, rabbitmq

6 months ago

To be clear, this was a request for org.fedoraproject.*.buildsync, not org.fedoraproject.*.buildsys. If that might be confusing, I'm open to other suggestions. Maybe rebuilder instead of buildsync?

Upon further reflection, I was also thinking that we should request a couple other topics while doing this. Could we add:

  • org.fedoraproject.*.rebuilder.eln.enqueue
  • org.fedoraproject.*.rebuilder.eln.batch.create,
  • org.fedoraproject.*.rebuilder.eln.batch.state_change,
  • org.fedoraproject.*.rebuilder.eln.attempt.create
  • org.fedoraproject.*.rebuilder.eln.attempt.state_change

This would allow us to set up notifications for batch rebuild state as well as retry attempts.

I actually missed the buildsync. In this case this will need to be a new fedora messaging schema.

I actually missed the buildsync. In this case this will need to be a new fedora messaging schema.

What can we expect for a timeframe?

@abompard I think setting up the new queue in rabbitmq shouldn't take much. Am I right?
@sgallagh You should provide the schema if this is a new publisher, but we already have few apps that are just publishing messages without schema, so it's not a hard requirement. See https://fedora-messaging.readthedocs.io/en/stable/user-guide/messages.html

PR #1610 should:
- allow the distrobuildsync-eln user to publish to any topic below the org.fedoraproject.{stg,prod}.rebuilder.eln. prefix
- subscribe the distrobuildsync-eln queue to the new topics you listed.

Is this what you wanted @sgallagh ?

PR #1610 should:
- allow the distrobuildsync-eln user to publish to any topic below the org.fedoraproject.{stg,prod}.rebuilder.eln. prefix
- subscribe the distrobuildsync-eln queue to the new topics you listed.

Is this what you wanted @sgallagh ?

Almost exactly! Though that revealed why we had a different bug that I had to work around in the ELNBuildSync code. We weren't receiving some notifications, but apparently that's because we needed to tell you which ones to send us. (I thought our client config handled that with its routing_keys setting).

I'll work on adding a message schema. I assume I follow the instructions at https://fedora-messaging.readthedocs.io/en/stable/tutorial/schemas.html and then send a merge request to the fedora-messaging repo?

Nope, the message schemas are a separate python package that you own, publish and update when you deem it necessary. Consumers of the messages will only install this package. Feel free to ask if you need any assistance with this, and/or if the docs aren't clear.

Nope, the message schemas are a separate python package that you own, publish and update when you deem it necessary. Consumers of the messages will only install this package. Feel free to ask if you need any assistance with this, and/or if the docs aren't clear.

I used koji-fedoramessaging-messages as inspiration and put together https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/elnbuildsync-messaging which I would welcome a review of, if you have a chance.

Sorry I missed your answer @sgallagh, I'll have a look at it today

Yeah it looks right! There are a few leftover references to Koji here and there but it's in comments and descriptions, so no harm done.

Login to comment on this ticket.

Metadata