Zodbot currently is not interacting with the new Fedora Accounts, so some functions are not working (e.g #9922).
The basis of this issue is that the supybot-fedora plugin that implements many of the fedora-related functions for zodbot needs to be updated. During fedora-accounts development, supybot-fedora was updated to python3, so it could use fasjson_client to talk to Fedora Accounts. However, the majority of the other plugins that make up zodbot are still python2, so we cannot just update easily.
This was investigated in #9922 -- check out that ticket for some background, but here is a list of the tasks that we will need to finish to get this mini-initiative done:
Metadata Update from @asaleh: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dev, medium-gain, medium-trouble, ops
Okay, started on moving supybot-koji from https://pagure.io/supybot-koji to fedora-infra/supybot-koji on github, and it seems the content in the pagure repo is an older version of supybot-fedora, not supybot-koji, so just going to import the source from the last koji build.
supybot-koji now is imported into:
https://github.com/fedora-infra/supybot-koji
Metadata Update from @ryanlerch: - Issue assigned to ryanlerch
Some random thoughts:
FWIW, I am not sure pinglists is still used much... we might just skip it and wait and see if anyone complains?
supybot-fedmsg I think is the thing that does the meetingminutes, so would be good to fix #9820 too. :)
Converting mote might be difficult, but if we can do it, great.
There is also another thing on value01... fedmsg-irc. This is the thing that makes all the irc bots that print out message events to irc channels, ie, fm-admin or fm-releng This needs to be ported to python3/fedora-messaging as well. ;(
If we wanted a quicker win here, I thought of another thing we could do: Make a NFS volume and copy /srv/meetbot from value01 to it. Make a rhel8 value02, move zodbot over to it and get everything up and running with python3 there. Leave mote and fedmsg-irc on value01 for now. value01 and value02 share /srv/meetbot via NFS.
That way we can move mote when it's done, move fedmsg-irc when it's done and in the mean time have zodbot moved more quickly. What do you think?
If we wanted a quicker win here, I thought of another thing we could do: Make a NFS volume and copy /srv/meetbot from value01 to it. Make a rhel8 value02, move zodbot over to it and get everything up and running with python3 there. Leave mote and fedmsg-irc on value01 for now. value01 and value02 share /srv/meetbot via NFS. That way we can move mote when it's done, move fedmsg-irc when it's done and in the mean time have zodbot moved more quickly. What do you think?
I think this might be the best approach -- that way we can focus on getting all the zodbot pieces in place.
supybot-fedora relies on sgmllib, there is a package in Fedora for 'python3-sgmllib3k' which supplies the dep we need, but it is not built for EPEL8. Here is a scratch build of it i did, we just need it in EPEL8
https://kojipkgs.fedoraproject.org//work/tasks/6262/68326262/python3-sgmllib3k-1.0.0-2.el8.noarch.rpm
Branch requested: https://pagure.io/releng/fedora-scm-requests/issue/34215
Thanks @kevin i also requested the following branches for EPEL8 too:
Missed supybot-fedmsg:
https://pagure.io/releng/fedora-scm-requests/issue/34221
oops. I forgot about that... we don't have fedmsg in epel8... nor do we particularly want to maintain it there. ;(
So, we need to either port to fedora-messging sooner rather than later, or switch and do fedora34/python3? :(
Sorry this stuff is so intertangled.
the latest commit by @abompard is "Migrate to Fedora Messaging" -- so this might already be done for us!
https://github.com/fedora-infra/supybot-fedmsg
might need to change the name of the plugin though :)
supybot-koji is now moved to github, converted to python3, and i cut a new 0.3 release of it.
https://github.com/fedora-infra/supybot-koji/releases/tag/0.3
and have built and checked that it works on el8. I added the new spec to to the github repo, but here is the scratchbuild:
https://koji.fedoraproject.org/koji/taskinfo?taskID=68584874
@kevin do you know what the supybot-notify plugin is used for (if anything?)
https://github.com/fedora-infra/supybot-notify
according to the readme it:
This plugin starts a TCP server on the given port and address, and forwards data sent the the server to an IRC channel.
supybot-meetbot is now moved from its old home on pagure.io into github:
https://github.com/fedora-infra/supybot-meetbot
i have applied the patches that were in the epel7 RPMs, and started the conversion to python3. Got the package building in el8, but ran into a few issues that are a bit out of my python capabilities -- so filed issues here:
https://github.com/fedora-infra/supybot-meetbot/issues
if we can fix these, pretty sure we will have a functioning meetbot on el8 / python3
The supybot-notify plugin is used for nagios. nagios sends alerts to zodbot via that for display in #fedora-noc.
DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3?
Will look into this today -- was hoping for the more seamless solution of a quick conversion -- but might be better to use the better fork.
Also, Is it possible to get these packages added to https://src.fedoraproject.org/group/infra-sig
supybot-fedmsg and supybot-fedora are already there at the moment.
Done.
DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3? Will look into this today -- was hoping for the more seamless solution of a quick conversion -- but might be better to use the better fork.
Ah i remember why i chose this route now -- supybot-fedmsg basically augments the methods in the original meetbot plugin -- was hoping to have minimal changes to it, but probably using the hcoop version is going to be the best.
Ah, if you think thats best we can do that. We don't Need the new version...
supybot-fedora update is now in bodhi for 0.5.0 in epel8:
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-406f547acd
Ok, so I packaged up hcoop-meetbot for Fedora, and it was all peachy -- then tried to build on EL8 -- hcoop-meetbot requires Python 3.7 :( (EL8 is 3.6 AFAIK)
so, back to https://github.com/fedora-infra/supybot-meetbot for now.
DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3? Ok, so I packaged up hcoop-meetbot for Fedora, and it was all peachy -- then tried to build on EL8 -- hcoop-meetbot requires Python 3.7 :( (EL8 is 3.6 AFAIK) so, back to https://github.com/fedora-infra/supybot-meetbot for now.
Ah man. ;( Thats anoying. ok. Lets go back to normal meetbot then...
Status Update
We now have the following plugins as projects in the fedora-infra github org, converted to python3, and workingand updates in flight in bodhi:
There is also an update for a dep (python-sgmllib3k) that is in the updates process for EPEL8 too:
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-dbe7362e3e
Next up is supybot-fedmsg, which is already in the fedora-infra github org, and converted to fedora-messaging, but needs converstion to python3, and packaging in EPEL8.
I have made a zodbot group in the fedora-infrastructure org on github:
https://github.com/orgs/fedora-infra/teams/zodbot/
all the repos that i created for this ticket are in there, just need to get the following two added somehow:
Okay, been trying to get supybot-fedmsg working, but to no avail -- since all this plugin does is add messages to supybot-meetbot, its probably easier and quciker to just implement the sending of the fedora messages into the meetbot plugin
so that is what i am going to try.
Okay, have now updated supybot-fedmsg with support for fedora messaging:
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-751e526302
now we don't have to use supybot-fedmsg
@kevin
I think that is all the plugins now updated and ready to try out in staging.
WRT getting supybot-meetbot to send fedora-messages, there is a new config item supybot.plugins.MeetBot.send_fedora_messaging: True (if we use the supybot based config for metbot), or send_fedora_messaging = True in meetingLocalConfig.py (if we configure meetbot that way)
supybot.plugins.MeetBot.send_fedora_messaging: True
send_fedora_messaging = True
Okay, also have cleaned up and added my test area -- introducing nonbot on tiny-stage https://github.com/fedora-infra/tiny-stage
Nonbot is pretty close to the live zodbot, but interacts with the other tinystage elements.
I have found it pretty useful when testing all these plugins out.
just a few quick config notes for when we deploy:
supybot.plugins.Fedora.fasjson.url
https://fasjson.fedoraproject.org/
supybot.plugins.Fedora.fas.password supybot.plugins.Fedora.fas.url supybot.plugins.Fedora.fas.username
So, I'm gonna try tomorrow to;
and then you can poke at the playbook to deploy the python3 ursabot to value02.stg? Once it's working enough, we do the same in prod.
Sound reasonable?
So, I'm gonna try tomorrow to; create a value02.stg (rhel8) create a nfs volume for meeting logs copy meeting contents over to it mount it on /srv/meetbot on value01.stg and value02.stg and then you can poke at the playbook to deploy the python3 ursabot to value02.stg? Once it's working enough, we do the same in prod. Sound reasonable?
Perfect. Thanks
Need to verify that IRC nicks from noggin work as expected when we deploy
https://pagure.io/fedora-infrastructure/issue/10021
the logic is there, just need to check that they work.
We are just waiting on the firewall to be opened up so zodbot stg (aka ursabot) can speak to the irc servers
The firewall should be open now (on port 7000 (ssl) to irc.libre.chat).
Fasntastic!
ursabot is now alive, and hangs out in #fedora-admin-stg
the meetbot part seems to work ok
the supybot-fedora plugin however is not working, as we are not authenticating with fasjson
This is what i added to value02 to get the keytab: https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/value.yml#_21
but it doent seem to be using it to auth with fasjson. I also tried adding the following to the ursabot.service file, but it didnt help :(
Environment=KRB5_CLIENT_KTNAME=/etc/krb5.ursabot_value02.stg.iad2.fedoraproject.org.keytab
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/supybot/files/ursabot.service
any thoughts?
the meetbot part seems to work ok the supybot-fedora plugin however is not working, as we are not authenticating with fasjson This is what i added to value02 to get the keytab: https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/value.yml#_21 but it doent seem to be using it to auth with fasjson. I also tried adding the following to the ursabot.service file, but it didnt help :( Environment=KRB5_CLIENT_KTNAME=/etc/krb5.ursabot_value02.stg.iad2.fedoraproject.org.keytab https://pagure.io/fedora-infra/ansible/blob/main/f/roles/supybot/files/ursabot.service any thoughts?
I had a quick look and the service keytab definitely works and can get user info from fasjson. I don't know how the app consumes the keytab to use though so I would think the issue may be there
Turns out it was a permissions issue on the keytab -- the ursabot service was being run as the daemon user, but the keytab permissions were for root.
this commit fixed this issue for me:
https://pagure.io/fedora-infra/ansible/c/1893bac187acb4c68bc0aad11ae5b229a1deecbc?branch=main
Have just noticed that ursabot is now refusing to connect to irc.libera.chat:7000 again (exactly the same errors I was seeing before the firewall was opened up)
could something have changef again with the firewall setup?
Also, before ursabot stopped connecting to libera again, there were a few additional issues with supybot-fedora when talking to fasjson that i fixed.
These fixes are in the new version 0.5.1 of supybot-fedora (please add karma if you can)
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-762ccf9ade
Have just noticed that ursabot is now refusing to connect to irc.libera.chat:7000 again (exactly the same errors I was seeing before the firewall was opened up) could something have changef again with the firewall setup?
I have pinged IT internally. I can reach irc.libera.chat on port 7000 from my local machine but not from staging so it looks as though it is a firewall issue.
Looks like libera.chat has shifted their ips around...
I asked IT to add:
but those ip's aren't in the current dns lookup. We could hard code one of them? or get IT to add more?
Looks like libera.chat has shifted their ips around... I asked IT to add: name: irc.libera.chat address: 172.106.11.86 name: irc.libera.chat address: 162.251.69.69 name: irc.libera.chat address: 162.251.69.69 but those ip's aren't in the current dns lookup. We could hard code one of them? or get IT to add more?
If they are using a CDN with variable addresses I'm not sure either will work as there is no knowing when the ips will change and what they will change to.
It's a network of donated servers, so it changes, but not constantly I don't think.
perhaps we should just ask them to allow port 700 outgoing to anything...
It's a network of donated servers, so it changes, but not constantly I don't think. perhaps we should just ask them to allow port 700 outgoing to anything...
Ah ok, in that case we could ask for a list of ips from them or just do 7000 to anywhere. Either will work I suppose
I hard coded an ip into ursabot config that had been allowed and it's back online. :)
We should still ask for 7000 anywhere, but this should at least unblock this...
awesome!~
everything seems to be working ok now -- meetings, getting "fasinfo" -- even though it is pulling from noggin.
not sure what the next steps here are... get more people to test ursabot?
also, not sure if mote is setup in staging to be able to test it out.
If it looks like ursabot is good to go, we should:
This way mote can move when it's ready and we don't have to wait for it... if that sounds good I can try and do the infra side stuff next week and you can get it ready on value02 and we can move it?
If it looks like ursabot is good to go, we should: deploy a rhel8 value02 in prod move /srv/meetbot to a netapp volume and mount on value01/02. stop zodbot on value01, sync anything we need/want to bring up zodbot on value02 This way mote can move when it's ready and we don't have to wait for it... if that sounds good I can try and do the infra side stuff next week and you can get it ready on value02 and we can move it?
Sounds good -- the only things we should need to do do on the zodbot side is tweak the config to use fasjson, and check that it authenticates with krb5 and works
ok, I have setup:
Can you:
Check and see if everything looks reasonable/right and either let me know what config to change for fasjson or perhaps schedule a time both of us can switch it over?
I think all thats left is: quit/disable zodbot on value01 sync /var/lib/zodbot again adjust for fasjson bring back up * close this and profit.
Oh, I guess we should test firewall and confirm we can connect from value02.
Looks like the playbook started it on value02... it ran for a few, but I stopped it.
From that you can see the logs on value02:
ERROR 2021-08-17T19:52:23 supybot Uncaught exception in Fedora.call: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/supybot/log.py", line 368, in m return f(self, args, *kwargs) File "/usr/lib/python3.6/site-packages/supybot/irclib.py", line 202, in call method(irc, msg) File "/usr/lib/python3.6/site-packages/supybot/plugins/Fedora/plugin.py", line 943, in doPrivmsg if word[-2:] in self.karma_tokens: AttributeError: 'Fedora' object has no attribute 'karma_tokens'
INFO 2021-08-17T19:53:55 supybot Received notice from NickServ on libera: "Invalid password for zodbot.".
etc
Of course as soon as we fix the password it's going to kill the other one... but at least this tells us what to fix.
Ok -- so thanks to @kevin zodbot in prod is now running on value02 / python3
for a bit there, karma was not working, due to the shelve/pickle db it uses needing upgrading from the python2 format to a python3 compatible format. (this was working in staging, because we started without a karmadb, and it created the right one out of the box)
a few things left not working properly:
.nextmeetings fails -- the calendar API url that is hardcoded in supybot-fedora is not working, needs to be updated to calendar.fedoraproject.org
fedora-messaging needs to be setup and enabled correctly on both staging and prod (i thought this was working in staging, but it actually wasnt sending the messages through to the queue).
.listmessages (part of the supybot-meetbot plugin) and most of the regular meetbot commands arent working. meetbot and #startmeeting etc etc is working just the additional commands that are run outside of meeting are not for some reason
THere is also a "send_fedora_messaging" config setting for meetbot that needs to be set to True
But i think the main issue here is that we need to get a new cert created for ursabot / zodbot for fedora messaging, and configrue them correctly
@ryanlerch is it possible to make fasinfo show the + for sponsor status again. Like used to stuff would show up in my list of groups like +sysadmin-noc if I was a sponsor for that group.
I created certs for zodbot (in prod) and ursabot (staging) in private...
ie, "rabbitmq/{{env}}/pki/"
Filed a bug over on supybot-fedora:
https://github.com/fedora-infra/supybot-fedora/issues/80
can you check i have the request correct over there?
Some other items today:
showticket was broken. I hotfixed the fix:
--- plugin.py.orig 2021-08-23 23:42:04.106602012 +0000 +++ plugin.py 2021-08-23 19:34:50.133050128 +0000 @@ -873,7 +873,7 @@ text = utils.web.getUrl(url, size=size) parser = Title() try: - parser.feed(text) + parser.feed(text.decode()) except sgmllib.SGMLParseError: irc.reply(format("Encountered a problem parsing %u", url)) if parser.title:
Also, meetbot when it finsihes a meeting gives:
[11:20:27] <zodbot> Meeting ended Mon Aug 23 18:20:27 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . [11:20:27] <zodbot> Minutes: fedora-meeting-3/2021/fedora-meeting-3.2021-08-23-18.00.html [11:20:27] <zodbot> Minutes (text): fedora-meeting-3/2021/fedora-meeting-3.2021-08-23-18.00.txt [11:20:27] <zodbot> Log: fedora-meeting-3/2021/fedora-meeting-3.2021-08-23-18.00.log.html
There should be a 'https://meetbot.fedoraproject.org/' inn front of all those links so they are full urls... but for some reason it's not showing up. ;(
and we still aren't getting any fedora-messages. ;(
Hmmm, not sure about the meeting finish message, but it is currently set to:
supybot.plugins.MeetBot.endMeetingMessage: Meeting ended %(endtime)s %(timeZone)s. Information about MeetBot at %(MeetBotInfoURL)s .\\nMinutes: %(urlBasename)s.html\\nMinutes (text): %(urlBasename)s.txt\\nLog: %(urlBasename)s.log.html
i think adding `%(logUrlPrefix)s' infront of the urlBasenames should fix this.
i dont have owner access, and i didnt want to turn off zodbot to just change that variable in the file :)
OKay, i think the URLs for endmeeting links are fixed now -- a change of config variables fixed it
things that rely on fedocal should now be fixed too -- we were using an old hardcoded fedocal link -- turned this into a config variable, and also there were some references to freenode hardcoded when talking about meeting locations -- this had been changed in the caledars, but not in supybot-fedora. so this is now changed.
.vacation, .nextmeeting(s) both use fedocal, so these are working again now.
.pushduty does too, but it seems to be disabled see #10187
anyhoo, i hotpatched the fixes on zodbot prod, but and the patch is commited to develop for supybot-fedora so we will pick it up when we release it again which should be soon.
So, is fedora-messaging working yet? or thats still to do?
Side note: we should kill the supybot-meetbot project on pagure.io if we are using the fedora-infra github one now.
No i have been trying to get fedora-messaging working on stage -- but to no avail. Is there someone i should ping for assistance here?
When i tested on tiny-stage it all works -- so i think it is something todo with authenticating to the stage / prod queues
Side note: we should kill the supybot-meetbot project on pagure.io if we are using the fedora-infra github one now. +1
+1
we should also archive supybot-fedmsg too -- as we dont use it anymore -- the new fedora-messaging stuff is now in the meetbot plugin
FWIW too, now i have set up the certs in ansible (for stage only for now), this is the error i am getting when trying to send a fedora-message to stage:
maybe i have the fedora-messaging config not right? i really am not sure.,
Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: ERROR 2021-08-27T09:34:28 Uncaught exception in MeetBot.__call__: Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: Traceback (most recent call last): Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/supybot/log.py", line 368, in m Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: return f(self, *args, **kwargs) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/supybot/irclib.py", line 202, in __call__ Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: method(irc, msg) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/supybot/plugins/MeetBot/plugin.py", line 123, in doPrivmsg Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: M.addline(nick, payload) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/supybot/plugins/MeetBot/meeting.py", line 528, in addline Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: linenum=linenum, time_=time_) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/supybot/plugins/MeetBot/meeting.py", line 278, in do_startmeeting Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: self.sendfedoramessage("meeting.start", **kwargs) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/supybot/plugins/MeetBot/meeting.py", line 583, in sendfedoramessage Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: fedora_messaging.api.publish(msg) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/fedora_messaging/api.py", line 315, in publish Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: eventual_result.wait(timeout=timeout) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib/python3.6/site-packages/crochet/_eventloop.py", line 241, in wait Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: result.raiseException() Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: File "/usr/lib64/python3.6/site-packages/twisted/python/failure.py", line 488, in raiseException Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: raise self.value.with_traceback(self.tb) Aug 27 09:34:28 value02.stg.iad2.fedoraproject.org supybot[2439760]: fedora_messaging.exceptions.ConnectionException
maybe
I'm not sure whats going on here.
@abompard @pingou @scoady can any of you help out with the fedora-messaging setup here?
This is causing mote to not update, when is gettning people antsi. ;(
Two more minor issues with meetbot:
1) The Log: line is missing the https://meetbot.fedoraproject.org/
2) The month-day are missing from the directory it makes. For example: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2021/ should be https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2021-08-30/
Two more minor issues with meetbot: 1) The Log: line is missing the https://meetbot.fedoraproject.org/
It looks like the config item for that is missing the prefix -- someone with access to config needs to update that
Yeah, not sure about this one -- the config line in the config file shows %%F for generating that directory, and it should be the YYYY-MM-DD format. on ursabot is does they year only, but the config item uses %Y so that is expected i suppose.
I'm not sure whats going on here. @abompard @pingou @scoady can any of you help out with the fedora-messaging setup here? This is causing mote to not update, when is gettning people antsi. ;(
Yeah, not sure why this isnt working -- and the error message i get back is not helpful -- def need some more assistance on the fedora-messaging side here
So, I looked at the directory thing. It seems it's using the default, even though we have correctly configured it as a conf variable. ;( I have hotfixed zodbot by overriding the default, but we should try and figure out why it's not using out config setting. ;(
#filenamePattern = '%(channel)s/%%Y/%(channel)s.%%F-%%H.%%M' filenamePattern = '%(channel)s/%%F/%(meetingname)s.%%F-%%H.%%M'
zodbot.conf:supybot.plugins.MeetBot.filenamePattern: %(channel)s/%%F/%(meetingname)s.%%F-%%H.%%M
I didn't see it mentioned here, but there are no messages in the meetingminutes archive since August 23. I'm assuming it's related.
Yes, thats the fedora-messaging not being sent thats causing that. ;(
@ryanlerch Can we setup a video call tomorrow where you show me where things are at? I may be able to help here
First quick look at the /etc/fedora-messaging/ursabot.toml on value01.stg, the rabbitmq server specified there is rabbitmq.stg.fedoraproject.org which resolves to wildcard, not a round robin of rabbitmq0{1,2,3} nor proxies, at least in staging.
/etc/fedora-messaging/ursabot.toml
value01.stg
rabbitmq.stg.fedoraproject.org
wildcard
rabbitmq0{1,2,3}
Thats expected. We use the cname 'wildcard' to mean 'the proxies'
So on value02.stg (note: value01 is rhel7/python2, value02 is rhel8/python3 and where the bots are active)
[root@value02 ~][STG]# host rabbitmq.stg.fedoraproject.org rabbitmq.stg.fedoraproject.org is an alias for wildcard.stg.fedoraproject.org. wildcard.stg.fedoraproject.org has address 10.3.166.74 wildcard.stg.fedoraproject.org has address 10.3.166.75
and those are the correct ips for proxy01.stg and proxy02.stg. So, it should reach the rabbitmq cluster fine. Also, other apps in stg appear working...
OK cool.
The RabbitMQ user in the configuration file (ursabot) did not exist on the server, so I created it and restarted the bot. The connection to Fedora Messaging seems to be working now, messages are sent on meeting starts and ends.
ursabot
The log URLs seem to be wrong though, it's not the full URL but only the path:
Minutes: fedora-admin-stg/2021/fedora-admin-stg.2021-09-16-08.39.html Minutes (text): fedora-admin-stg/2021/fedora-admin-stg.2021-09-16-08.39.txt Log: fedora-admin-stg/2021/fedora-admin-stg.2021-09-16-08.39.log.html
The log URLs seem to be wrong though, it's not the full URL but only the path: Minutes: fedora-admin-stg/2021/fedora-admin-stg.2021-09-16-08.39.html Minutes (text): fedora-admin-stg/2021/fedora-admin-stg.2021-09-16-08.39.txt Log: fedora-admin-stg/2021/fedora-admin-stg.2021-09-16-08.39.log.html
I think this is fixed now on production (its a config setting). I didnt bother setting it on ursabot, as we dont have a staging mote, so there is nowhere to point it at
OK cool. The RabbitMQ user in the configuration file (ursabot) did not exist on the server, so I created it and restarted the bot. The connection to Fedora Messaging seems to be working now, messages are sent on meeting starts and ends.
@kevin is it worth doing a FBR to get this fixed on prod? or just wait until freeze is over? It appears to be sending the messages as expected now on staging. thanks @abompard
IMHO, yes. Actually I think it might count as an outage and just need to be fixed, but you can do a freeze break if you prefer. ;)
Do we know if mote is picking up the messages ok in stg? https://meetbot.stg.fedoraproject.org ? also, does it seem to be sending to meeting-minutes? I guess that may be disabled in stg tho... (we do have a stg mote at that address. ;)
So did we fail to have a call to the rabbitmq role to make a user?
Anyhow, +1 to fixing it asap.
@nirik
IMHO, yes. Actually I think it might count as an outage and just need to be fixed, but you can do a freeze break if you prefer. ;) Do we know if mote is picking up the messages ok in stg?
Do we know if mote is picking up the messages ok in stg?
I tried to check this, but the meeting logs on stg dont seem to be shared between value01.stg and value02.stg like it is on prod? Was there a NFS mount we are using in prod for this that isnt on stage?
Anyhoo, i looked at the fedmsg consumer on mote, and it likely will not work -- we will need to tweak this like to add a .v1 to the end of the topic it is expecting:
https://github.com/fedora-infra/mote/blob/master/mote/fedmsg_consumer.py#L31
but i will try to fix zodbot so it sends the messages, and then hotfix mote to see if it works as expected.
ok -- made the changes on prod to try to get it sending the messages, and it doenst seem to be working.
there isnt any errors in the logs, so not sure how to begin troubleshooting this.
Ok, it probably helps to use the correct config file:
https://pagure.io/fedora-infra/ansible/pull-request/797#request_diff
will try to see if this works.
OK. we are sending fedora messages now.
I hotpatched the meetbot plugin to remove the .v1 from the topic, so we dont need to change that anymore in Mote. (will update the plugin code to reflect this)
mote should be running the update script now everytime it gets a meeting ending message.
Not sure what is going on with FMN though -- it should be matching and sending emails to the meetingminutes list, but its not working for some reason.
is it a possibility that we can send the emails directly from the meetbot plugin? rather than dealing with FMN?
Awesome. Thanks!
The meetingminutes thing might just be due to FMN lag... it's like 14 hours behind right now. ;(
I suppose we could look at just sending directly. Thats not great because it means we have an app that sends emails and FMN was supposed to be the single point for that...
It did technically send an email (on a delay), but it's....not great. https://lists.fedoraproject.org/archives/list/meetingminutes@lists.fedoraproject.org/thread/AH7OYTRG63TOGCHWZYTX5MGNKFQOFYBX/
ok, meetbot is now sending better messages that doesnt break FMN -- the emails to the meetingminutes mailing list are there and properly formatted again now.
also fixed the underlying issue with the log URLs and the date log location issues, so this is working properly now. this is the PR for these changes:
https://github.com/fedora-infra/supybot-meetbot/pull/19
i have hotfixed zodbot and ursabot with these changes, so will need to spin up a new release at some point.
awesome. Shall we keep this open for that release being installed and close then? Or should we just close now. I don't think there's anything else left.
Many thanks for working on this!
Just spun up a new release of supybot-meetbot
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-e681cb1233
Not sure if we have to keep this ticket open, but id like to get all the old repos closed / archived / deleted now too -- just so i don'd confuse myself when we loop back to this in 3 months time :)
Also, i think the meeting logs in ursabot arent being synced between value01 and value02 like on prod -- not sure where in the ansible setup this is located, so couldnt really debug it
+1. I already closed the pagure supybot-meetbot. Let me know what other ones need to be closed... or any that you can't just do yourself. ;)
Yeah, they are not. I was hoping this would be somewhat temporary and mote would move to python3 or be replaced soon, so I didn't setup staging storage. If it's helpfull I can do so?
Okay, here are the remaining repos to archive / delete:
i see https://github.com/fedora-infra/supybot-meetbot is removed -- now i have the old histroy imported into github, we can do the same for https://pagure.io/meetbot
ok. All those should be done. ;)
One last thing... I see some errors in emails still:
Subject: Cron apache@value01 /usr/local/bin/meetings_by_team.sh
ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ambassadors_latam_2017_dic_22th/ambassadors_latam_2017_dic_22th.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ambassadors_latam_meeting/ambassadors_latam_meeting.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_azure_working_group/ansible_azure_working_group.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_contributor_summit_brooklyn/ansible_contributor_summit_brooklyn.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_core_irc_meeting/ansible_core_irc_meeting.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_core_irc_public_meeting/ansible_core_irc_public_meeting.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_core_meeting_https:github/ansible_core_meeting_https:github.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_core_team_irc_meeting/ansible_core_team_irc_meeting.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_core_team/ansible_core_team.tgz’: File exists ln: failed to create symbolic link ‘/srv/web/meetbot/teams/ansible_cwg_sprint/ansible_cwg_sprint.tgz’: File exists ...
and indeed those exist... but its running with a ln -sf, so it should force the link?
I opened a new ticket on that issue, so I am gonna go ahead and close this now.
Many thanks ryan! if there's anything more to do we missed, please re-open or file a new ticket.
zodbot is now running on value02.iad2.fedoraproject.org with python3 on rhel8.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Issue status updated to: Open (was: Closed)
Login to comment on this ticket.