RHEL 7 EOL is approaching (June 2024) and we still have some services running on RHEL7 machines.
github2fedmsg is of these services and we need to do something about it.
Current state of things:
ARC investigation document
Recommendation was to replace it with more general solution webhook2fedmsg
webhook2fedmsg
June 2024
Metadata Update from @zlopez: - Issue assigned to abompard
This is submitted as outreachy project with @abompard as potential mentor.
The outreachy project started this week. @ghostle is the one who will be working on webhook2fedmsg :thumbsup:
Checking in on progress as we're close to the expected done date.
The outreachy project isn't close to being usable at this stage, we may need to keep github2fedmsg running for a couple months, if it's not too much of a problem. Otherwise me or @ryanlerch can try to port it quickly (and partially) to make the main part work on RHEL9 until the replacement is ready.
I'd really prefer we move it off rhel7... if you could do a quick port of it that would be nice.
Note that it might currently be broken to adding new projects: https://pagure.io/fedora-infrastructure/issue/11911
the main issue here with a quick 1:1 port is that it is using the velruse library (very old and unmaintained) to create webhooks (and other stuff) on github.
What we could in theory do, however, is disable the web interface, and just listen for existing webhook calls from the projects that already have it set up.
I'm for disabling the web interface, it didn't work well for some time anyway. We just need to probably replace the page with some info, why it's not working anymore.
It actually does work... the problem reported was a issue with permissions on the github side. ;(
Is there any way we can see how many new repos have been added when? if there's not too many being added, I'd be ok just announcing it will be blocked for a short time. Or perhaps we could come up with an interem way to manually enable? ie, ask people just to file tickets until we get the web side working again?
Any update? :-)
We should really push this now and announce that we are getting the github2fedmsg down. We should announce it on announce list and by small blog post.
@abompard Do you want to take care of that?
bump
@zlopez : I'm still investigating a Mirrormanager issues, do you mind doing the blog post please?
@abompard Let me prepare something
The announcement of github2fedmsg sunset was sent to:
The sunset date was announced to 3rd February, first Monday in February.
Preparing for decommission of github2fedmsg and fedmsg on Monday. I spent today looking for the mentions of fedmsg in various places and here is the list. I'm not sure about removing items marked with ?. @kevin @abompard Could you check?
?
Repositories: * https://github.com/fedora-infra/github2fedmsg * https://github.com/fedora-infra/fedmsg Documentation: * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/fedmsg-relay/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/fedmsg-gateway/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/fedmsg-introduction/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/fedmsg-new-message-type/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/anitya/#_fedmsg_messages_arent_sent * (mention) https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/fedmsg-websocket/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/pdc/ * (mention) https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/odcs/ * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/mbs/ * (mention) https://docs.fedoraproject.org/en-US/infra/sle_services/ * (mention) https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/rabbitmq/ * (mention) https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/ansible/#_logging * https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/openqa/#_autocloud_resultsdb_forwarder_autocloudreporter * https://docs.fedoraproject.org/en-US/infra/release_guide/workflow_automation/ ? Inventory: * github2fedmsg01.iad2.fedoraproject.org * github2fedmsg01.stg.iad2.fedoraproject.org * busgateway01.iad2.fedoraproject.org * busgateway01.stg.iad2.fedoraproject.org Ansible: * files/communishift/haproxy.cfg (mention) * filter_plugins/fedmsg.py * handlers/restart_services (tasks) * inventory/group_vars/all (variables) * inventory/group_vars/autosign (variables) * inventory/group_vars/batcave * inventory/group_vars/bodhi_backend (variables) * inventory/group_vars/bodhi_backend_stg (variables) * inventory/group_vars/buildhw (mention) * inventory/group_vars/buildvm (mention) * inventory/group_vars/buildvm_aarch64 (mention) * inventory/group_vars/buildvm_aarch64_stg (mention) * inventory/group_vars/buildvm_ppc64le (mention) * inventory/group_vars/buildvm_ppc64le_stg (mention) * inventory/group_vars/buildvm_s390x (mention) * inventory/group_vars/buildvm_s390x_stg (mention) * inventory/group_vars/buildvm_stg (mention) * inventory/group_vars/busgateway * inventory/group_vars/busgateway_stg * inventory/group_vars/github2fedmsg * inventory/group_vars/github2fedmsg_stg * inventory/group_vars/greenwave ? * inventory/group_vars/greenwave_stg ? * inventory/group_vars/koji (variables) * inventory/group_vars/koji_stg (variables) * inventory/group_vars/mailman (variables) * inventory/group_vars/mailman_stg (variables) * inventory/group_vars/nagios (variables) * inventory/group_vars/notifs_backend ? * inventory/group_vars/notifs_backend_stg ? * inventory/group_vars/notifs_web ? * inventory/group_vars/notifs_web_stg ? * inventory/group_vars/openqa_onebox_test ? * inventory/group_vars/pagure (variables) * inventory/group_vars/pagure_stg (variables) * inventory/group_vars/people (variables) * inventory/group_vars/pkgs (variables) * inventory/group_vars/pkgs_stg (variables) * inventory/group_vars/proxies (mention) * inventory/group_vars/proxies_stg (mention) * inventory/group_vars/releng_compose (variables) * inventory/group_vars/retrace (variables) * inventory/group_vars/staging (variables) * inventory/group_vars/value (variables) * inventory/group_vars/value_stg (variables) * inventory/group_vars/waveirdb (variables) * inventory/group_vars/waveirdb_stg (variables) * inventory/group_vars/wiki (variables) * inventory/group_vars/wiki_stg (variables) * inventory/host_vars/bodhi-backend01.iad2.fedoraproject.org (variables) * inventory/host_vars/bodhi-backend01.stg.iad2.fedoraproject.org (variables) * inventory/host_vars/compose-branched01.iad2.fedoraproject.org (variables) * inventory/host_vars/compose-iot01.iad2.fedoraproject.org (variables) * inventory/host_vars/compose-rawhide01.iad2.fedoraproject.org (variables) * inventory/host_vars/compose-x86-01.iad2.fedoraproject.org (variables) * inventory/host_vars/compose-x86-01.stg.iad2.fedoraproject.org (variables) * inventory/host_vars/people01.fedoraproject.org (variables) * inventory/inventory * playbooks/fedmsgupdate.yml * playbooks/groups/busgateway.yml * playbooks/groups/github2fedmsg.yml * playbooks/groups/value.yml (fedmsg-service) * playbooks/include/proxies-miscellaneous.yml (fedmsg roles) * playbooks/include/proxies-reverseproxy.yml (github2fedmsg) * playbooks/manual/restart-fedmsg-services.yml * playbooks/manual/uninstall_ipa_client.yml (github2fedmsg) * playbooks/manual/upgrade/datanommer.yml (fedmsg-hub) * playbooks/manual/upgrade/fedmsg.yml * playbooks/manual/upgrade/packages.yml (fedmsg-hub) * playbooks/openshift-apps/messaging-bridges.yml * roles/abrt/faf/defaults/main.yml (mention) * roles/abrt/faf/templates/fedora-messaging.toml.j2 (mention) * roles/base/templates/iptables/iptables (fedmsg) * roles/base/templates/iptables/iptables.bastion (fedmsg) * roles/base/templates/iptables/iptables.staging (fedmsg) * roles/base/templates/iptables/iptables.torrent02.fedoraproject.org (fedmsg) * roles/base/templates/nftables/nftables (fedmsg) * roles/base/templates/nftables/nftables.staging (fedmsg) * roles/base/templates/nftables/nftables.torrent02.fedoraproject.org (fedmsg) * roles/batcave/tasks/main.yml (comment fedmsg) * roles/batcave/templates/fedora-messaging.toml.j2 (mention) * roles/bugzilla2fedmsg/tasks/main.yml (comment fedmsg) * roles/bugzilla2fedmsg/templates/bugzilla2fedmsg.ini (comment fedmsg) * roles/collectd/base/files/fedmsg-types.db * roles/collectd/base/files/fedmsg.conf * roles/collectd/base/tasks/main.yml (fedmsg - busgateway) * roles/collectd/fedmsg-activation * roles/collectd/fedmsg-service * roles/copr/backend/templates/copr-be.conf.j2 (comment fedmsg) * roles/distgit/pagure/files/stunnel.service (fedmsg docs mention) ? * roles/distgit/pagure/templates/fedora-messaging.toml (mention) * roles/fedmsg * roles/git/hooks/files/post-receive-chained * roles/git/hooks/files/post-receive-chained-forks * roles/git/hooks/files/post-receive-fedmsg * roles/github2fedmsg * roles/gitolite/check_fedmsg_hooks ? * roles/haproxy/templates/haproxy.cfg (fedmsg, github2fedmsg) * roles/hosts/files/koji01.stg.iad2.fedoraproject.org-hosts (github2fedmsg) * roles/ipsilon/templates/configuration.conf (mention) * roles/koji_hub/templates/fedora-messaging.toml (mention) * roles/messaging/base/templates/messaging.toml.j2 (comment fedmsg) * roles/nagios_client/files/scripst/check_fedmsg_consumer_backlog.py * roles/nagios_client/files/scripst/check_fedmsg_consumer_exceptions.py * roles/nagios_client/files/scripst/check_fedmsg_producer_last_ran.py * roles/nagios_client/files/scripst/check_fedmsg_producers_consumers.py * roles/nagios_client/tasks/main.yml (fedmsg) * roles/nagios_client/templates/check_fedmsg_composer_proc.cfg.j2 * roles/nagios_client/templates/check_fedmsg_consumers.cfg.j2 * roles/nagios_client/templates/check_fedmsg_gateway_proc.cfg.j2 * roles/nagios_client/templates/check_fedmsg_hub_proc.cfg.j2 * roles/nagios_client/templates/check_fedmsg_irc_proc.cfg.j2 * roles/nagios_client/templates/check_fedmsg_relay_proc.cfg.j2 * roles/nagios_server/files/nagios/commands/notify.cfg (fedmsg command) * roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-ircbot.cfg (fedmsg) * roles/nagios_server/files/nagios/contacts/fedmsg.cfg * roles/nagios_server/files/nagios/plugins/check_fedmsg_consumer_backlog.py * roles/nagios_server/files/nagios/plugins/check_fedmsg_consumer_exceptions.py * roles/nagios_server/files/nagios/plugins/check_fedmsg_producers_consumers.py * roles/nagios_server/files/nagios/services/iad2_internal/fedmsg.cfg * roles/nagios_server/templates/nagios/services/websites.cfg.j2 (github2fedmsg) * roles/nagios_server/templates/nrpe/nrpe.cfg.j2 (fedmsg checks) * roles/openshift-apps/badges/templates/fm-fedbadges.toml (mention) * roles/openshift-apps/badges/templates/fm-tahrir.toml (mention) * roles/openshift-apps/coreos-koji-tagger/templates/deploymentconfig.yml.j2 (github2fedmsg mention) * roles/openshift-apps/datanommer/templates/fedora-messaging.toml (mention) * roles/openshift-apps/elections/templates/fedora-messaging.toml (mention) * roles/openshift-apps/fedocal/templates/fedora-messaging.toml (mention) * roles/openshift-apps/fmn/templates/api.toml (mention) * roles/openshift-apps/kerneltest/templates/fedora-messaging.toml (mention) * roles/openshift-apps/messaging-bridges * roles/openshift-apps/mirrormanager/templates/fedora-messaging.toml.j2 (mention) * roles/openshift-apps/noggin/templates/fedora-messaging.toml (mention) * roles/openshift-apps/poddlers/templates/fedora-messaging.toml.j2 (mention) * roles/openshift-apps/release-monitoring/templates/anitya.toml (logging config) * roles/openshift-apps/webhook2fedmsg/templates/fedora-messaging.toml.j2 (mention) * roles/packages3 ? * roles/pagure/files/stunnel.service (fedmsg docs mention) ? * roles/pagure/tasks/main.yml (fedmsg-relay) * roles/pagure/templates/fedora-messaging.toml (mention) * roles/rabbit/queue/tasks/main.yml (fedmsg bridges) * roles/supybot/templates/fedora-messaging.toml.j2 (mention) * main.yml (fedmsg, github2fedmsg)
Preparing for decommission of github2fedmsg and fedmsg on Monday. I spent today looking for the mentions of fedmsg in various places and here is the list. I'm not sure about removing items marked with ?. @kevin @abompard Could you check? ``` https://docs.fedoraproject.org/en-US/infra/release_guide/workflow_automation/ ?
```
I think this is an old doc and can be removed.
roles/distgit/pagure/files/stunnel.service (fedmsg docs mention) ?
I think this can be removed, and also the commented out section in: roles/distgit/pagure/tasks/main.yml that refers to it.
roles/gitolite/check_fedmsg_hooks ?
This entire roles/gitolite is no longer used and can be removed.
roles/packages3 ?
This can be removed. It was replaced I think by fedora-packages-static in openshift. Should also remove mentions to the role.
roles/pagure/files/stunnel.service (fedmsg docs mention) ?
I think that needs to stay? Not sure why we keep it on pagure.io but remove on src.fp.o? That should be the thing that updates tickets when you are still on the page and someone makes a change.
There is also:
playbooks/include/proxies-miscellaneous.yml
which has a call to role: fedmsg/gateway/slave that should be removed.
Also:
roles/openshift-apps/messaging-bridges playbooks/openshift-apps/messaging-bridges.yml
and the app should be deleted on openshift and rm -rf /etc/openshift_apps/messaging_bridges removed from os-control01{stg}
As the CoreOS team needs more time to move over from github2fedmsg we postponed the date of decommission to 13th February. I will still start decommission in staging now and wait with production till the new date.
I sent postpone announcement to both devel-announce and infra lists, so people are aware. I will also add comment to announcement blog post.
Started with cleaning the docs. Here is the PR.
I didn't updated https://pagure.io/infra-docs-fpo/blob/master/f/modules/releng_misc_guide/pages/sop_pushing_updates.adoc. It has plenty of fedmsg-hub mentions, but I didn't want to just remove them. It needs somebody with the knowledge to update this doc.
The messaging-bridges openshift project and github2fedmsg VM are now gone on staging. I created cleaning PR for staging in ansible.
What remains:
So the PR is now in place. I removed also gitolite and packages3 role, as those two were tied to fedmsg.
I'm just wondering what to do with roles/openshift-apps/monitor-dashboard/ as that is using fedmsg as well. @kevin Do we want to keep it?
roles/openshift-apps/monitor-dashboard/
fedmsg
Nope, see https://pagure.io/fedora-infrastructure/issue/12230 we want to retire it.
I can do that as part of this work as it will not work anymore without fedmsg.
So I did the sunset today:
What is missing:
I will keep this open till the last remaining things are resolved, but the github2fedmsg is now gone.
The nagios alert is now gone.
Metadata Update from @zlopez: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.