#8213 fedmsg -> fedora-messaging migration tracker
Opened a year ago by kevin. Modified 2 months ago

We should track and finish moving everything from fedmsg to fedora-messaging.

We have at least:

releng/scripts/fedmsg-functions.sh (used by branched/rawhide composes)
roles/nagios_server/files/nagios/commands/notify.cfg (used by nagios)

and all the following playbooks still calling fedmsg/base:

playbooks/groups/autocloud-backend.yml:  - fedmsg/base
playbooks/groups/autocloud-web.yml:  - fedmsg/base
playbooks/groups/badges-backend.yml:  - fedmsg/base
playbooks/groups/badges-web.yml:  - fedmsg/base
playbooks/groups/bugyou.yml:  - fedmsg/base
playbooks/groups/bugzilla2fedmsg.yml:  - fedmsg/base
playbooks/groups/busgateway.yml:  - fedmsg/base
playbooks/groups/datagrepper.yml:  - fedmsg/base
playbooks/groups/elections.yml:  - fedmsg/base
playbooks/groups/fas.yml:  - fedmsg/base
playbooks/groups/fedimg.yml:  - fedmsg/base
playbooks/groups/fedocal.yml:  - fedmsg/base
playbooks/groups/freshmaker.yml:  - fedmsg/base
playbooks/groups/github2fedmsg.yml:  - fedmsg/base
playbooks/groups/hubs.yml:  - fedmsg/base
playbooks/groups/kerneltest.yml:   - fedmsg/base
playbooks/groups/koji-hub.yml:  - fedmsg/base
playbooks/groups/koschei-backend.yml:  - fedmsg/base
playbooks/groups/loopabull.yml:    - fedmsg/base
playbooks/groups/mailman.yml:  - fedmsg/base
playbooks/groups/mbs.yml:  - fedmsg/base
playbooks/groups/mdapi.yml:  - fedmsg/base
playbooks/groups/mirrormanager.yml:  - fedmsg/base
playbooks/groups/modernpaste.yml:  - fedmsg/base
playbooks/groups/noc.yml:  - fedmsg/base
playbooks/groups/notifs-backend.yml:  - fedmsg/base
playbooks/groups/notifs-web.yml:  - fedmsg/base
playbooks/groups/nuancier.yml:  - fedmsg/base
playbooks/groups/odcs.yml:  - fedmsg/base
playbooks/groups/odcs.yml:  - role: fedmsg/base
playbooks/groups/openqa.yml:   - { role: fedmsg/base, tags: ['fedmsg_base', 'fedmsg'] }
playbooks/groups/packages.yml:  - fedmsg/base
playbooks/groups/pdc.yml:  - fedmsg/base
playbooks/groups/pdc.yml:  - fedmsg/base
playbooks/groups/people.yml:  - fedmsg/base
playbooks/groups/pkgs.yml:  - fedmsg/base
playbooks/groups/releng-compose.yml:  - fedmsg/base
playbooks/groups/resultsdb.yml:   - { role: fedmsg/base,
playbooks/groups/retrace.yml:# - fedmsg/base
playbooks/groups/sundries.yml:  - role: fedmsg/base
playbooks/groups/taskotron.yml:   - { role: fedmsg/base }
playbooks/groups/value.yml:  - fedmsg/base
playbooks/groups/wiki.yml:  - fedmsg/base
playbooks/groups/zanata2fedmsg.yml:  - fedmsg/base
playbooks/hosts/happinesspackets-stg.fedorainfracloud.org.yml:  - fedmsg/base
playbooks/hosts/happinesspackets.fedorainfracloud.org.yml:  - fedmsg/base

Koschei migration from fedmsg to fedora-messaging in production is planned on 2019-09-26. Staging instance already uses fedora-messaging, eg. this message ("username": "amqp-bridge")

@karsten you were going to look at the infrastructure scripts moving right? Could you perhaps focus on/start with planet? It seems to somehow be broken right now, and I can't seem to figure out why. Moving it to fedora-messaging would hopefully fix it and get us a bit further along too at the same time. :)

It runs on people02.fedoraproject.org... the venus-planet package will need adjusting I guess: https://koji.fedoraproject.org/koji/buildinfo?buildID=942792

Or if thats not your cup o tea, we can look for someone else. :)

Also, we need a plan for notifs.

playbooks/groups/resultsdb.yml: - { role: fedmsg/base,

I thought that we already ported this to fedora-messaging.

playbooks/groups/taskotron.yml: - { role: fedmsg/base }

As taskotron is in maintenance mode, I'm not really keen on putting time into porting the one sub-component which still uses fedmsg to fedora-messaging unless there's no other option

playbooks/groups/resultsdb.yml: - { role: fedmsg/base,

I thought that we already ported this to fedora-messaging.

We did, this is likely just a clean up that we didn't do :(

Same for elections, koji-hub and pkgs I think.

Also Ansible callback plugins need to be ported:

callback_plugins/fedmsg_callback.py
callback_plugins/fedmsg_callback2.py

Metadata Update from @cverna:
- Issue tagged with: backlog

a year ago

Metadata Update from @kevin:
- Issue assigned to karsten

a year ago

I'm dropping the following from the list:

autocloud retires with f29:

playbooks/groups/autocloud-backend.yml: - fedmsg/base
playbooks/groups/autocloud-web.yml: - fedmsg/base

we are giving away badges:

playbooks/groups/badges-backend.yml: - fedmsg/base
playbooks/groups/badges-web.yml: - fedmsg/base

bugyou never took off/should be retired now.

playbooks/groups/bugyou.yml: - fedmsg/base

elections we are giving away:

playbooks/groups/elections.yml: - fedmsg/base

fedocal we are giving away:

playbooks/groups/fedocal.yml: - fedmsg/base

hubs is gone.

playbooks/groups/hubs.yml: - fedmsg/base

going away:

playbooks/groups/modernpaste.yml: - fedmsg/base
playbooks/groups/nuancier.yml: - fedmsg/base

done:
playbooks/groups/koji-hub.yml: - fedmsg/base
playbooks/groups/pkgs.yml: - fedmsg/base
playbooks/groups/koschei-backend.yml: - fedmsg/base
playbooks/groups/openqa.yml: - { role: fedmsg/base, tags: ['fedmsg_base', 'fedmsg'] }
playbooks/groups/resultsdb.yml: - { role: fedmsg/base,

Unknown????????????

playbooks/groups/fedimg.yml: - fedmsg/base
playbooks/groups/freshmaker.yml: - fedmsg/base

That leaves:

playbooks/groups/bugzilla2fedmsg.yml: - fedmsg/base
playbooks/groups/busgateway.yml: - fedmsg/base
playbooks/groups/datagrepper.yml: - fedmsg/base
playbooks/groups/fas.yml: - fedmsg/base
playbooks/groups/github2fedmsg.yml: - fedmsg/base
playbooks/groups/kerneltest.yml: - fedmsg/base
playbooks/groups/loopabull.yml: - fedmsg/base
playbooks/groups/mailman.yml: - fedmsg/base
playbooks/groups/mbs.yml: - fedmsg/base
playbooks/groups/mdapi.yml: - fedmsg/base
playbooks/groups/mirrormanager.yml: - fedmsg/base
playbooks/groups/noc.yml: - fedmsg/base
playbooks/groups/notifs-backend.yml: - fedmsg/base
playbooks/groups/notifs-web.yml: - fedmsg/base
playbooks/groups/odcs.yml: - fedmsg/base
playbooks/groups/odcs.yml: - role: fedmsg/base
playbooks/groups/packages.yml: - fedmsg/base
playbooks/groups/pdc.yml: - fedmsg/base
playbooks/groups/pdc.yml: - fedmsg/base
playbooks/groups/people.yml: - fedmsg/base
playbooks/groups/releng-compose.yml: - fedmsg/base
playbooks/groups/retrace.yml:# - fedmsg/base
playbooks/groups/sundries.yml: - role: fedmsg/base
playbooks/groups/taskotron.yml: - { role: fedmsg/base }
playbooks/groups/value.yml: - fedmsg/base
playbooks/groups/wiki.yml: - fedmsg/base
playbooks/groups/zanata2fedmsg.yml: - fedmsg/base
ansible callbacks

playbooks/groups/bugzilla2fedmsg.yml: - fedmsg/base

Being ported by @abompard

playbooks/groups/fas.yml: - fedmsg/base

going away (sooner or later :()

playbooks/groups/kerneltest.yml: - fedmsg/base

Being handed over to Jeremy

playbooks/groups/loopabull.yml: - fedmsg/base

Ported to fedora-messaging upstream, we can migrate when we want

playbooks/groups/packages.yml

I think @cverna was saying yesterday that this no longer works, and thus we should be able to remove it.

Is there any sense of what should be prioritized higher in this list? Just curious for some expert opinion @kevin @pingou

There are a few angles and I don't think any good answer to this question:

  • We want to retire fedmsg so all of these should be ported to fedora-messaging
  • We will need to migrate datagrepper, busgateway and FMN from fedmsg to fedora-messaging but both of them require the features that today fedmsg_meta_fedora_infrastructure provides and these features are now handled as part of the message schemas that each application should provide
  • To make the migration from fedmsg faster, we have not added support to message schemas to the application we've migrated, so we'll need to go back to a number of our applications to add these schemas.

Some that we could look at porting today would be:

  • wiki
  • sundries (not quite sure what uses fedmsg there though)
  • retrace
  • odcs
  • mbs
  • people (here as well, not sure what uses fedmsg there)
  • releng-compose
  • noc

Once these are done we'll likely ran into the more complex ones.

There are a few angles and I don't think any good answer to this question:

We want to retire fedmsg so all of these should be ported to fedora-messaging
We will need to migrate datagrepper, busgateway and FMN from fedmsg to fedora-messaging but both of them require the features that today fedmsg_meta_fedora_infrastructure provides and these features are now handled as part of the message schemas that each application should provide
To make the migration from fedmsg faster, we have not added support to message schemas to the application we've migrated, so we'll need to go back to a number of our applications to add these schemas.

Some that we could look at porting today would be:

wiki
sundries (not quite sure what uses fedmsg there though)

I am not sure here either. It's using fedmsg/base, but might be only for the user id or something.

retrace
odcs
mbs
people (here as well, not sure what uses fedmsg there)

It's fedoraplanet. It emits a fedmsg on blog posts.

releng-compose
noc

Once these are done we'll likely ran into the more complex ones.

We should consider doing some design work on the replacement for FMN and datagrepper, since I think they will take the longest because we have to design new ones that avoid the mistakes of the current ones.

There's been some work going on by @jcline for a FMN replacement, I'm going to pick it up. About Datagrepper, my guess is that we could port it pretty easily by just changing the listener and ignoring message headers at first, but if there are design issues with the current one that we want to iron out, then yeah it's the right moment to think about a new design. Is there a document or a page that list the current issues with datanommer / datagrepper?

@karsten you were going to look at the infrastructure scripts moving right? Could you perhaps focus on/start with planet? It seems to somehow be broken right now, and I can't seem to figure out why. Moving it to fedora-messaging would hopefully fix it and get us a bit further along too at the same time. :)
It runs on people02.fedoraproject.org... the venus-planet package will need adjusting I guess: https://koji.fedoraproject.org/koji/buildinfo?buildID=942792
Or if thats not your cup o tea, we can look for someone else. :)

Just a note here that planet is in fact sending fedmsgs again. I have no idea what happened to fix it, but perhaps it's not the highest priority now. (I thought I noted this here, but I guess not, many appologies).

Also, if planet is proving hard we could revisit the software we use for that (if there's any better ones now) or some other way forward.

There's been some work going on by @jcline for a FMN replacement, I'm going to pick it up. About Datagrepper, my guess is that we could port it pretty easily by just changing the listener and ignoring message headers at first, but if there are design issues with the current one that we want to iron out, then yeah it's the right moment to think about a new design. Is there a document or a page that list the current issues with datanommer / datagrepper?

Not that I know of, but there are some that I think we could/should solve:

  • Right now the db grows without bound. It has every fedmsg since we turned it on in it. It's up to 492GB currently.
  • The db just stores every message, one per row, might be we could be more clever about how we store them so we could search them easier.
  • I think it is also python2...

After learning about them for koji, I think we might want to look at partitioning the main messages table. Split it out per month or something, then make sure queries default to just the current partition unless you specifically want to query older ones. This would make both inserts and queries a lot faster.

Additionally, we may want to talk to internal RH folks about this as I think they running a datanommer/datagrepper set and might be interested in our changes and wanting to move to them.

So, nothing super radical, but good to address.

I haven't found any retrace messages in datagrepper, either I'm using the wrong category or retrace doesn't emit anything.

So, retrace server is kind of an umbrella for 2 related projects:

  1. abrt - this is the thing that allows users to report crash backtraces to bugzilla. It's not used too much anymore. I don't think it emits fedmsgs at all.
  2. faf - https://retrace.fedoraproject.org/faf/summary/ - fedora analisys framework. This is a automatic crash reporter thing that gathers stats on crashes. It does emit fedmsgs.

I am not sure who the current developers are on faf/abrt/retrace. @msuchy is the admin contact. @msuchy who should we ask about porting this app to fedora-messaging?

I think perhaps the compose scripts might be highest priority here. Those are run from pungi-fedora and the nightly.sh script.

Why is playbooks/groups/busgateway.yml on this list ? Afaik thats for setting up the fedmsg relay and will get deleted once everything got moved to fedora-messaging / rabbitmq ?
Is there anything else that needs to be done here ?

Yeah, you are right, busgateway just goes away when fedmsg does. I got the initial list by looking at what playbooks called any fedmsg role in ansible...

I've posted patches to fedora-infra, waiting to get them merged before this can be closed

I think this got merged, right ? I don't have ACL to close this

Yeah, thanks for the patches!

Did we get all the applications? Or are there still some to move?

Metadata Update from @cverna:
- Assignee reset

5 months ago

Metadata Update from @pingou:
- Issue tagged with: dev

4 months ago

We've set this board: https://github.com/orgs/fedora-infra/projects/7 to track progress on adding fedora-messaging message schemas to apps.

I'm looking forward the day we'll do some serious clean up on our ansible repo :)

I've distilled information from this ticket and packages we have in Fedora (including -infra tags) into a summary doc. I'm not sure I've caught everything, though :wink:. Many of the items I found also need to be triaged, i.e. whether an item is ours to take care of, someone else's or should go away.

@kevin @pingou @abompard, would you please give it a quick look over, correct mistakes you find and fill in info you have (e.g. move items into their correct places)? Thanks!

Thanks for doing that!

I moved freshmaker, zanata2fedmsg and packages to dead... packages we need to figure out what we are replacing it with, the others are already gone.

kerneltest we moved the old one as the new re-write one needs some more work before its ready. We will want to talk to justin about that.

@nphilipp I've looked at it and moved some things around but I'm a little confused on the difference between that doc and the board at: https://github.com/orgs/fedora-infra/projects/7 which is ordered more or less by priority and includes some of the items you've listed there (though it also misses some items that you've listed in hackmd I think).

@pingou as I understand it (@abompard correct me if I'm wrong), the board tracks the "has fedora-messaging, needs schemas" items, i.e. it was missing at least some items that need to be ported from fedmsg to fedora-messaging.

I started the document initially to compile the list of items that still need porting, then thought that when I'm touching each one of these, I can put in all the info I find, too. I.e. it should describe the initial status of hopefully all items we need to look at, so we can say e.g. "this item needs to be ported and schemas added" or "the schemas of this item need to be completed". it's not supposed to track work like the board, but should help us verify that we track all the items we should in it. Does that make sense?

Mind, I'm not sure if we should extend the scope of the board to cover porting work, too, or add a separate one for that.

Login to comment on this ticket.

Metadata
Boards 1
dev Status: Backlog