#9964 Update Zodbot to Python3
Opened 2 months ago by ryanlerch. Modified a day ago

Zodbot currently is not interacting with the new Fedora Accounts, so some functions are not working (e.g #9922).

The basis of this issue is that the supybot-fedora plugin that implements many of the fedora-related functions for zodbot needs to be updated. During fedora-accounts development, supybot-fedora was updated to python3, so it could use fasjson_client to talk to Fedora Accounts. However, the majority of the other plugins that make up zodbot are still python2, so we cannot just update easily.

This was investigated in #9922 -- check out that ticket for some background, but here is a list of the tasks that we will need to finish to get this mini-initiative done:

  • Create a repo for supybot-pinglists in the fedora-infra github, with the source from the last build in koji, convert to python3, and rebuild the rpm in the infra tag. Not doing this, pinglists is not really in use, so going to deprecate it from zodbot
  • move the supybot-koji, move the supybot-meetbot, and supybot-notify repos from pagure.io to the fedora-infra github, convert to python3, and rebuild the rpms. These 2 repos appear to have been added to pagure when fedorahosted closed up shop, but havent been touched since. Moving these to github where all the other zodbot plugin sources live makes sense IMHO.
  • check that supybot-fedmsg is python 3
  • convert mote to fedora-messaging and move zodbot and mote from value01.iad to a new value02.iad host running RHEL8 sideline this, and workaround using @kevin 's solution outlined in the comment below

Metadata Update from @asaleh:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: dev, medium-gain, medium-trouble, ops

2 months ago

Okay, started on moving supybot-koji from https://pagure.io/supybot-koji to fedora-infra/supybot-koji on github, and it seems the content in the pagure repo is an older version of supybot-fedora, not supybot-koji, so just going to import the source from the last koji build.

Metadata Update from @ryanlerch:
- Issue assigned to ryanlerch

2 months ago

Some random thoughts:

FWIW, I am not sure pinglists is still used much... we might just skip it and wait and see if anyone complains?

supybot-fedmsg I think is the thing that does the meetingminutes, so would be good to fix #9820 too. :)

Converting mote might be difficult, but if we can do it, great.

There is also another thing on value01... fedmsg-irc. This is the thing that makes all the irc bots that print out message events to irc channels, ie, fm-admin or fm-releng This needs to be ported to python3/fedora-messaging as well. ;(

If we wanted a quicker win here, I thought of another thing we could do:
Make a NFS volume and copy /srv/meetbot from value01 to it.
Make a rhel8 value02, move zodbot over to it and get everything up and running with python3 there.
Leave mote and fedmsg-irc on value01 for now.
value01 and value02 share /srv/meetbot via NFS.

That way we can move mote when it's done, move fedmsg-irc when it's done and in the mean time have zodbot moved more quickly. What do you think?

If we wanted a quicker win here, I thought of another thing we could do:
Make a NFS volume and copy /srv/meetbot from value01 to it.
Make a rhel8 value02, move zodbot over to it and get everything up and running with python3 there.
Leave mote and fedmsg-irc on value01 for now.
value01 and value02 share /srv/meetbot via NFS.

That way we can move mote when it's done, move fedmsg-irc when it's done and in the mean time have zodbot moved more quickly. What do you think?

I think this might be the best approach -- that way we can focus on getting all the zodbot pieces in place.

supybot-fedora relies on sgmllib, there is a package in Fedora for 'python3-sgmllib3k' which supplies the dep we need, but it is not built for EPEL8. Here is a scratch build of it i did, we just need it in EPEL8

https://kojipkgs.fedoraproject.org//work/tasks/6262/68326262/python3-sgmllib3k-1.0.0-2.el8.noarch.rpm

Thanks @kevin i also requested the following branches for EPEL8 too:

oops. I forgot about that... we don't have fedmsg in epel8... nor do we particularly want to maintain it there. ;(

So, we need to either port to fedora-messging sooner rather than later, or switch and do fedora34/python3?
:(

Sorry this stuff is so intertangled.

the latest commit by @abompard is "Migrate to Fedora Messaging" -- so this might already be done for us!

https://github.com/fedora-infra/supybot-fedmsg

might need to change the name of the plugin though :)

supybot-koji is now moved to github, converted to python3, and i cut a new 0.3 release of it.

https://github.com/fedora-infra/supybot-koji/releases/tag/0.3

and have built and checked that it works on el8. I added the new spec to to the github repo, but here is the scratchbuild:

https://koji.fedoraproject.org/koji/taskinfo?taskID=68584874

@kevin do you know what the supybot-notify plugin is used for (if anything?)

https://github.com/fedora-infra/supybot-notify

according to the readme it:

This plugin starts a TCP server on the given port and address, and forwards
data sent the the server to an IRC channel.  

supybot-meetbot is now moved from its old home on pagure.io into github:

https://github.com/fedora-infra/supybot-meetbot

i have applied the patches that were in the epel7 RPMs, and started the conversion to python3. Got the package building in el8, but ran into a few issues that are a bit out of my python capabilities -- so filed issues here:

https://github.com/fedora-infra/supybot-meetbot/issues

if we can fix these, pretty sure we will have a functioning meetbot on el8 / python3

The supybot-notify plugin is used for nagios. nagios sends alerts to zodbot via that for display in #fedora-noc.

DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3?

DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3?

Will look into this today -- was hoping for the more seamless solution of a quick conversion -- but might be better to use the better fork.

Also, Is it possible to get these packages added to https://src.fedoraproject.org/group/infra-sig

  • supybot-notify
  • supybot-koji
  • supybot-meetbot

supybot-fedmsg and supybot-fedora are already there at the moment.

DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3?

Will look into this today -- was hoping for the more seamless solution of a quick conversion -- but might be better to use the better fork.

Ah i remember why i chose this route now -- supybot-fedmsg basically augments the methods in the original meetbot plugin -- was hoping to have minimal changes to it, but probably using the hcoop version is going to be the best.

Ah, if you think thats best we can do that. We don't Need the new version...

DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3?

Ok, so I packaged up hcoop-meetbot for Fedora, and it was all peachy -- then tried to build on EL8 -- hcoop-meetbot requires Python 3.7 :( (EL8 is 3.6 AFAIK)

so, back to https://github.com/fedora-infra/supybot-meetbot for now.

DId we want to just look at that other fork of meetbot? It looks like they also added some more features as well as moving everything to python3?

Ok, so I packaged up hcoop-meetbot for Fedora, and it was all peachy -- then tried to build on EL8 -- hcoop-meetbot requires Python 3.7 :( (EL8 is 3.6 AFAIK)

so, back to https://github.com/fedora-infra/supybot-meetbot for now.

Ah man. ;( Thats anoying. ok. Lets go back to normal meetbot then...

Next up is supybot-fedmsg, which is already in the fedora-infra github org, and converted to fedora-messaging, but needs converstion to python3, and packaging in EPEL8.

I have made a zodbot group in the fedora-infrastructure org on github:

https://github.com/orgs/fedora-infra/teams/zodbot/

all the repos that i created for this ticket are in there, just need to get the following two added somehow:

  • supybot-fedmsg
  • supybot-fedora

Okay, been trying to get supybot-fedmsg working, but to no avail -- since all this plugin does is add messages to supybot-meetbot, its probably easier and quciker to just implement the sending of the fedora messages into the meetbot plugin

so that is what i am going to try.

Okay, have now updated supybot-fedmsg with support for fedora messaging:

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-751e526302

now we don't have to use supybot-fedmsg

@kevin

I think that is all the plugins now updated and ready to try out in staging.

WRT getting supybot-meetbot to send fedora-messages, there is a new config item supybot.plugins.MeetBot.send_fedora_messaging: True (if we use the supybot based config for metbot), or send_fedora_messaging = True in meetingLocalConfig.py (if we configure meetbot that way)

Okay, also have cleaned up and added my test area -- introducing nonbot on tiny-stage https://github.com/fedora-infra/tiny-stage

Nonbot is pretty close to the live zodbot, but interacts with the other tinystage elements.

I have found it pretty useful when testing all these plugins out.

just a few quick config notes for when we deploy:

  • for fedora-supybot we will need to set up krb5 access for fasjson so the plugin can query the user data, then set the new supybot.plugins.Fedora.fasjson.url config value to https://fasjson.fedoraproject.org/ -- while there, we can set these config values to nothing too, since we wont use them anymore:
supybot.plugins.Fedora.fas.password
supybot.plugins.Fedora.fas.url
supybot.plugins.Fedora.fas.username
  • As said before, the meetbot plugin now does the work for supybot-fedmsg -- so we can disable supybot-fedmsg, and enable fedora-messaging with supybot.plugins.MeetBot.send_fedora_messaging: True
    *supybot-pinglists wasnt ported as discussed above, so we will need to not install that anymore, and disable it from supybot

So, I'm gonna try tomorrow to;

  • create a value02.stg (rhel8)
  • create a nfs volume for meeting logs
  • copy meeting contents over to it
  • mount it on /srv/meetbot on value01.stg and value02.stg

and then you can poke at the playbook to deploy the python3 ursabot to value02.stg?
Once it's working enough, we do the same in prod.

Sound reasonable?

So, I'm gonna try tomorrow to;

  • create a value02.stg (rhel8)
  • create a nfs volume for meeting logs
  • copy meeting contents over to it
  • mount it on /srv/meetbot on value01.stg and value02.stg

and then you can poke at the playbook to deploy the python3 ursabot to value02.stg?
Once it's working enough, we do the same in prod.

Sound reasonable?

Perfect. Thanks

Need to verify that IRC nicks from noggin work as expected when we deploy

https://pagure.io/fedora-infrastructure/issue/10021

the logic is there, just need to check that they work.

We are just waiting on the firewall to be opened up so zodbot stg (aka ursabot) can speak to the irc servers

The firewall should be open now (on port 7000 (ssl) to irc.libre.chat).

Fasntastic!

ursabot is now alive, and hangs out in #fedora-admin-stg

the meetbot part seems to work ok

the supybot-fedora plugin however is not working, as we are not authenticating with fasjson

This is what i added to value02 to get the keytab:
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/value.yml#_21

but it doent seem to be using it to auth with fasjson. I also tried adding the following to the ursabot.service file, but it didnt help :(

Environment=KRB5_CLIENT_KTNAME=/etc/krb5.ursabot_value02.stg.iad2.fedoraproject.org.keytab

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/supybot/files/ursabot.service

any thoughts?

the meetbot part seems to work ok

the supybot-fedora plugin however is not working, as we are not authenticating with fasjson

This is what i added to value02 to get the keytab:
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/value.yml#_21

but it doent seem to be using it to auth with fasjson. I also tried adding the following to the ursabot.service file, but it didnt help :(

Environment=KRB5_CLIENT_KTNAME=/etc/krb5.ursabot_value02.stg.iad2.fedoraproject.org.keytab

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/supybot/files/ursabot.service

any thoughts?

I had a quick look and the service keytab definitely works and can get user info from fasjson. I don't know how the app consumes the keytab to use though so I would think the issue may be there

Turns out it was a permissions issue on the keytab -- the ursabot service was being run as the daemon user, but the keytab permissions were for root.

this commit fixed this issue for me:

https://pagure.io/fedora-infra/ansible/c/1893bac187acb4c68bc0aad11ae5b229a1deecbc?branch=main

Have just noticed that ursabot is now refusing to connect to irc.libera.chat:7000 again (exactly the same errors I was seeing before the firewall was opened up)

could something have changef again with the firewall setup?

Also, before ursabot stopped connecting to libera again, there were a few additional issues with supybot-fedora when talking to fasjson that i fixed.

These fixes are in the new version 0.5.1 of supybot-fedora (please add karma if you can)

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-762ccf9ade

Have just noticed that ursabot is now refusing to connect to irc.libera.chat:7000 again (exactly the same errors I was seeing before the firewall was opened up)

could something have changef again with the firewall setup?

I have pinged IT internally. I can reach irc.libera.chat on port 7000 from my local machine but not from staging so it looks as though it is a firewall issue.

Looks like libera.chat has shifted their ips around...

I asked IT to add:

  • name: irc.libera.chat
    address: 172.106.11.86
  • name: irc.libera.chat
    address: 162.251.69.69
  • name: irc.libera.chat
    address: 162.251.69.69

but those ip's aren't in the current dns lookup. We could hard code one of them? or get IT to add more?

Looks like libera.chat has shifted their ips around...

I asked IT to add:

  • name: irc.libera.chat
    address: 172.106.11.86
  • name: irc.libera.chat
    address: 162.251.69.69
  • name: irc.libera.chat
    address: 162.251.69.69

but those ip's aren't in the current dns lookup. We could hard code one of them? or get IT to add more?

If they are using a CDN with variable addresses I'm not sure either will work as there is no knowing when the ips will change and what they will change to.

It's a network of donated servers, so it changes, but not constantly I don't think.

perhaps we should just ask them to allow port 700 outgoing to anything...

It's a network of donated servers, so it changes, but not constantly I don't think.

perhaps we should just ask them to allow port 700 outgoing to anything...

Ah ok, in that case we could ask for a list of ips from them or just do 7000 to anywhere. Either will work I suppose

I hard coded an ip into ursabot config that had been allowed and it's back online.
:)

We should still ask for 7000 anywhere, but this should at least unblock this...

Login to comment on this ticket.

Metadata
Boards 3
ops Status: Backlog
dev Status: Backlog
mini-initative Status: In Progress