#9576 Please update discourse2fedmsg to use fedora messaging and deploy in infrastructure
Opened 9 months ago by mattdm. Modified 17 days ago

Describe what you would like us to do:


discourse2fedmsg is a small (currently 80 lines of python) service (a flask app) which bridges the Discourse forum to our message bus. I'd like this running in production somewhere, taking messages from both discussion.fedoraproject.org and ask.fedoraproject.org

I assume that this needs to actually be converted from fedmsg to the new message bus.

There may also be some adjustments needed to get the messages into a useful format. This was running in test before but never in production

When do you need this to be done by? (YYYY/MM/DD)


I would love to have this up and running before the next Flock -- well, Nest with Fedora. So, sometime like 2021/06/01 would be ideal.

It is not urgent, but I would love to be able to add forum participation to our general community metrics. It would also allow us to add Fedora Badges for forum activity.


This sounds more like a initiative to me: https://docs.fedoraproject.org/en-US/cpe/initiatives/
But we have one similar initiative this quarter, so we will see if this could be part of it.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: dev, high-gain, high-trouble, ops

9 months ago

Metadata Update from @zlopez:
- Issue priority set to: Waiting on External (was: Waiting on Assignee)

9 months ago

Oh, also: this is the current fedmsg service: https://pagure.io/discourse2fedmsg. Patrick had it deployed in a testing instance but it is no longer running.

Also: we would want to be able to distinguish between different discourse instances. I'm not sure if the current code does this.

Metadata Update from @pingou:
- Issue tagged with: request-for-resources

9 months ago

Metadata Update from @kevin:
- Issue tagged with: mini-iniative

7 months ago

I would be nice to port the code to fedora-messaging before we deploy it to prod. It'll simplify the ansible playbooks.

One question I should answer (with your help!) is: what events do we want to translate to fedmsg. Here's the options

Screenshot_2021-09-16_at_18-57-04_Admin_-_Ask_Fedora.png

My thinking is that new topics and posts being created is obviously useful, and "solved" could be used for badges and other things (at least if it carries the name of the person who posted the solution -- otherwise less so). The discourse badges are separate from Fedora Badges, and I don't know what to do about that. Possibly completely disable them, but one thing we could do is use this to echo them instead.

Group stuff we don't need because that's gonna come from the SSO, and other stuff is basically management blah blah blah (categories, etc.)

Maybe "likes" is good -- likes are not anonymous on the site, so it's not revealing anything to broadcast them. Could be useful and indicates reading/lurking activity.

And I don't think user events is really particularly useful except for stalking, which... is a non-goal.

One question I should answer (with your help!) is: what events do we want to translate to fedmsg. Here's the options

Screenshot_2021-09-16_at_18-57-04_Admin_-_Ask_Fedora.png

My thinking is that new topics and posts being created is obviously useful, and "solved" could be used for badges and other things (at least if it carries the name of the person who posted the solution -- otherwise less so). The discourse badges are separate from Fedora Badges, and I don't know what to do about that. Possibly completely disable them, but one thing we could do is use this to echo them instead.

Group stuff we don't need because that's gonna come from the SSO, and other stuff is basically management blah blah blah (categories, etc.)

Maybe "likes" is good -- likes are not anonymous on the site, so it's not revealing anything to broadcast them. Could be useful and indicates reading/lurking activity.

And I don't think user events is really particularly useful except for stalking, which... is a non-goal.

These all look good to me. I think the only thing we could possibly use User events for is aggregate data to track overall activity on the site but I'm not sure if this is available elsewhere

These all look good to me. I think the only thing we could possibly use User events for is aggregate data to track overall activity on the site but I'm not sure if this is available elsewhere

Yeah, I'm a little torn. The site does have its own reporting tools, but there's something amazing about having all of this in the Fedora message stream.

On the balance, though, I think that for those purposes, measuring topics, posts, and likes gives us the most useful view into meaningful activity anyway.

Also, I should say this directly in case it's unclear -- we'll want to distinguish Ask Fedora and Discussion posts in the messages. That's probably inherent, but just to be sure.

Also, I should say this directly in case it's unclear -- we'll want to distinguish Ask Fedora and Discussion posts in the messages. That's probably inherent, but just to be sure.

I spend some time today to look at what needs to be done with the code.

The code itself is pretty simple, it's just waiting till anything arrives to the /webhook endpoint, verifies if it's an event from discourse and then sends the JSON payload as fedmsg message with topic taken from X-Discourse-Event-Type.

Here is the list of things, that needs to be done:

  • Port over to fedora messaging

  • Create fedora messaging schema (This could be tricky because we don't know what could be in the payload, but maybe after looking at the discourse documentation this will be clear)

  • Create unit tests (at least some simple tests would be nice, maybe adding format and lint checker would be a nice bonus)

  • Update README.md (current one is too brief)

  • [Nice to have] CI

I created new issues on the discourse2fedmsg tracker based on the list I in previous comment.
You can see them on discourse2fedmsg tracker.

Login to comment on this ticket.

Metadata
Boards 3
dev Status: Backlog
ops Status: Backlog
mini-initative Status: Backlog