@ngompa you had started looking at the packaging for EL8 no?

Yes, I've started working on this for almost all of our software. Mailman is already in Fedora, but I don't know where our HyperKitty and Postorius packages are...

@ngompa out of curiosity, have you had a chance to look where it's at now?

I've started looking. I think I found this: https://abompard.fedorapeople.org/current-mailman-hyperkitty-deps/

Is this what we're using? Or is it somewhere else?

@abompard d you know? :)

Taking info from associate ticket

present deployed seems to be:

Postorius Documentation • GNU Mailman • Postorius Version 1.1.2

upstream has released up to 1.2.3

https://postorius.readthedocs.io/en/latest/news.html

Describe what you need us to do:

Update to a later version, as there is a an 'across panels' leak of data when two mailing lists are open and an update occurs in one ... an async status message is displayed in all panels

When do you need this? (YYYY/MM/DD)

no deadline -- nice to have

When is this no longer needed or useful? (YYYY/MM/DD)

no expiration date

If we cannot complete your request, what is the impact?

a cross site exploit seems to exist

to demonstrate unsubscribe from the following mailing lists

For usage: https://lists.fedorahosted.org/archives/list/firewalld-users@lists.fedorahosted.org/

For development: https://lists.fedorahosted.org/archives/list/firewalld-devel@lists.fedorahosted.org/

then close the browser to get a fresh cache state (I use Firefox latest in CentOS)

open two tabs

choose one and enter a subscription transaction

change to the other tab

(the XSS leak is in a green box up top, saying:
Please check your inbox for further instructions
)

it appears in both tabs

(From #8629 ) GitHub recently sent us this message:

Hi @FedoraInfra,

On February 6th, 2020 at 00:07 (UTC) your application (Fedora Mailman) used an access token (with the User-Agent python-requests/2.9.1) as part of a query parameter to access an endpoint through the GitHub API:

https://api.github.com/user

Please use the Authorization HTTP header instead as using the `access_token` query parameter is deprecated.

Depending on your API usage, we'll be sending you this email reminder once every 3 days for each token and User-Agent used in API calls made on your behalf.
Just one URL that was accessed with a token and User-Agent combination will be listed in the email reminder, not all.

Visit https://developer.github.com/changes/2019-11-05-deprecated-passwords-and-authorizations-api/#authenticating-using-query-parameters for more information.

Thanks,
The GitHub Team

It's likely that the library that does social authentication in HyperKitty needs to be upgraded to use more recent APIs.

I'm working on updating the packages. Hopefully I'll have some progress in the coming days.

I'll track packages and their status here: https://hackmd.io/@nilsph/SyBqxUVK8

As discussed on IRC, building current versions of the missing packages in COPR (and Koji as the case may be). Tracking status here.

Documenting caveats (differences to the old deployment etc.) here.

Documenting caveats (differences to the old deployment etc.) here.

I believe we have a way of shipping alternative versions of pytest... @churchyard might know something here.

But we could also simply just not run the tests on EPEL8. Not sure if we want to do that, given how much modifications we're doing...

I've been summoned. Reading the ticket now...

You can create a "compat" package with pytest4 in EPEL. People will love you for that. I can help.

Or you can use the Python 3.8 module for newer pytest, but that is another can of worms, because you don't have other deps.

Or you can figure out why does pytest-django actually need >=3.6 and have that fixed.

Let me know which way you'd like to proceed.

@churchyard In order of preference:

1. Have pytest-django use older pytest
2. pytest4 compat package
3. py38 module

I don't think using the py38 module is realistic because we'd need to rebuild everything and adjust the packaging for it and deal with module building...

Package pytest-django 3.2.1?

Alternatively, revert this https://github.com/pytest-dev/pytest-django/commit/d9a9e9a8866d3966ad44f89b6244aec70580afb1 (but there might be more adjustments needed)

Personally, I'd prefer a somewhat current pytest4 compat package over packaging version 3.2.x of pytest-django (which seems to be incompatible with newer pytest versions, i.e. potentially obstruct the way forward). I concur with staying away from a Python 3.8 based stack if we can avoid it (though it would intuitively mesh well with "here be current versions").

@churchyard, how would I go about such a compat package? If I'm not off track, it wouldn't be parallel-installable to the existing version, so wouldn't need special treatment in terms of file conflicts etc. Would it simply get a namespaced (S)RPM name, e.g. pytest4, python3-pytest4 with the proper Python provides so consumers like pytest-django could then do s.th. like BuildRequire: %{py3_dist pytest} >= 3.6 to get the compat package?

Pretty much what you say. I can help. In fact, if we get the python-pytest4 component, we might end up reusing it as a "lower" version compat package in Fedora, because stuff is incompatible with pytest 5. I just don't want to update to pytest 5 before the 3.9 bootstrap to avoid having to bootstrap both of them. I can submit the package for review and we can build it first in epel8 only, and after the 3.9 bootstrap, also build it in Fedora when updating pytest to 5.

Sounds good?

Sounds good?

I'm game, sounds good.

I'd need comaintainers for the epel8 branch.

Count me in.

Feel free to add @infra-sig. Since we need it for infra apps, it's just better to make sure everyone can maintain it.

OK, after some digging, this won't work:

• We cannot have compat packages in epel that conflict with rhel
• A conflict in a non-modular package with a modular package would make the thing explode

Sorry.

Feel free to add @infra-sig. Since we need it for infra apps, it's just better to make sure everyone can maintain it.

👍

@nphilipp I think we're going to have to go with trying to get things to work with older pytest on EPEL8. The other strategies seem basically unworkable.

@duck let's coordinate reviewing your (OSPO's) more up to date set of packages here. As a reference for others, here's your COPR and associated repo which I understand to be preliminary work for the "reviewing pipeline":

• https://copr.fedorainfracloud.org/coprs/g/pizza-cats/mailman3-rpm-repo-api-3.1/
• https://gitlab.com/osas/mailman3-rpm-pkg

Can you add me to the respective groups so I can try my luck at updating & building the packages? My account on GitLab is @nilsph. Not sure if it's strictly needed, I could just put my changes into another repo but I guess it'd be good to have everything in one place.

@duck My name is @Conan_Kudo there as well...

I added you to the GL project, please only make changes to the api-3.1 branch (but you can make your own), so that it does not affect our production.

As for Copr, if you wish to be able to trigger builds could you make a request to be a member? (I cannot add people in the UI, it works the other way around only).

@duck I guess I have to be a member of the pizza-cats FAS group to work with the COPR repo, but according to FAS, the group's invite only. As an admin of the group, the group page should show you an "Add User" field where you can put in my FAS user name (nphilipp).

The pizza-cats group is for the OSPO Comminfra Team, to manage all our resources (even if right now I did not migrate them all yet), so I cannot use this group.

But you can make a request for a Copr Project and I will accept it. Once loggued in if you go to the project's Settings you can check for Request Permissions.

The Other way around is to create a new FAS group for the Mailman maintenance that we could use (I suppose) in src.fedoraproject.org too. It would be nice to have proper group maintenance on all resources.

So I finally got mailman3 to build fine again. Interestingly Click decided to uselessly change its quoting it broke the tests and I had to backport a recent version for EL8. So now it's fine on both rawhide and EL8. I'm going to create a PR on the official package.

python-mailman-client requires vcrpy and pytest-services for its test suite so that's my next steps.

The pizza-cats group is for the OSPO Comminfra Team, to manage all our resources (even if right now I did not migrate them all yet), so I cannot use this group.

Understood.

But you can make a request for a Copr Project and I will accept it. Once loggued in if you go to the project's Settings you can check for Request Permissions.

Ah, wasn't aware of that. I've requested build permissions there.

The Other way around is to create a new FAS group for the Mailman maintenance that we could use (I suppose) in src.fedoraproject.org too. It would be nice to have proper group maintenance on all resources.

I think for taking care of stuff in koji, bodhi, Bugzilla, it's fine if we just use the existing groups, i.e. infra-sig on our end (I assume you have something comparable on the OSPO side).

I think for taking care of stuff in koji, bodhi, Bugzilla, it's fine if we just use the existing groups, i.e. infra-sig on our end (I assume you have something comparable on the OSPO side).

That's the goal of the newly created pizza-cats group, but limited to our custom stuff needed for our communities.

Should I join the infra-sig group?

Currently the mailman3 package and possibly some deps (I don't think he ever uploaded hyperkitty and postorius) are owned by abompard and it would be nice to get a change of ownership since he won't be involved anymore. I guess the python sig is taking care too, and some deps might not be useful anymore, thus some cleanup might be needed.

I hit a build issue and reported a bug to Copr:
https://bugzilla.redhat.com/show_bug.cgi?id=1848938
Not reply yet but that's write annoying (or should I say blocking).
If you have any idea, please tell me or comment on the ticket.

Could someone review the mailman3 PR in the meantime?
https://src.fedoraproject.org/rpms/mailman3/pull-request/4

I've finally managed to fix mailman3 package in Fedora again: https://bodhi.fedoraproject.org/updates/FEDORA-2020-9d4ef73a14

Next up: Postorius and HyperKitty

thanks @ngompa . last time I had time to work on this I made changes in this package for python-mailman-client and there were problems because python3-zope-i18nmessageid was not provided anymore by any of the dependencies. The fact that Fedora is a very fast moving target is really a pain honestly.

The problem I see with just bumping the version to an unreleased version is that we have no idea if this will work properly with Hyperkitty and Postorius. Versions need to be coordinated and that's a combination that's not supported upstream. Also this build is not tested on EL8, which is our (OSPO) target. I would have preferred backporting patches but that's way more work.

OK, we're lucky, the API version is still 3.1, so inter-communication between the components should work. Now we can see if the build of other components work too.

It would be nice to have all those packages in the main fedora repo when the time comes.

That's the goal! :wink:

Know where a issue is to do that with the rest of the Fedora Web infrastructure? That would make it very convenient to update to the latest. (and safer, too)

So far Fedora has been too fast a moving target for me to handle this alone. Also there seem to be a confusion between being able to build and having the set all working properly together. Last time I checked mailman-client's tests were failing so that mean the software is not going to work well.

The real struggle is our patches on the stack. Porting those is painful. Unfortunately, I can't drop some of them, because they deal with non-upstream functionality and alternative Python dependencies. :frowning:

How hard is it to get those upstream?

Well I made it so hyperkitty can be updated for lists.fedoraproject.org some one just has to make a project on Fedora.

Currently it is 404: https://src.fedoraproject.org/rpms/HyperKitty

Thoughts?

The main problematic patch that hasn't been upstreamed yet is this one: https://gitlab.com/mailman/mailman/-/merge_requests/721

Looks like this is still pending upstream. ;(

@ngompa is there a checklist of things people could help with here? aside from the patch?

Getting HyperKitty and Postorius packaged would be great. That can be done while I work through Mailman itself.

Quack,

In the past at OSPO we started packaging a newer version, continuing Aurelien Bompard's work further to the Python 3 world. We do not plan to use the fast-moving Fedora target but EL-8 for stability but we wanted to collaborate on Fedora integration. This has proved just too much and Fedora was really changing too fast and I spent more timr trying to patch deps that kept breaking than finishing packaging the rest of the deps tree. Also when EL8 was just out so many packages were missing. I opened bugs but most were unanswered and when the packages were finally there the BR were not closed and I had no idea it was available. Nils Philippsen proposed to help and we started organizing but soon after he vanished without a trace. So when other projects came I just simply quit.

I'm trying to see if I could reboot the project now but this time I'm going to focus on EL8 only to focus on our production but if anyone wants to collaborate, you're welcome.

Our packaging repo is here:
https://gitlab.com/osas/mailman3-rpm-pkg
and we use Copr to build them.

Most notably you'll find postorius and hyperkitty packages from Aurelien and updated to fix some problems. I'm having a look at newer version to see how the deps evolved.

_o<

Ironically, working on EL8 was killing me because the Mailman project focuses on supporting newer components rather aggressively, which is why I focused on Fedora.

I agree this is a problem, especially because Python 3.6 is rather old now and even when a project advertise 3.6+ certain base components like setuptools or pytests in the distro are not recent enough. This attempt may fail indeed.

I 'm not sure what to do about that but upgrading all the instances we manage for various communities every so often is clearly not something I'm looking for either. Having packages that build fine is not sufficient to ensure the service is going to work well; I'm not following closely but I hope there are plans for runtime tests (I read something about reusing autopkgtest in the past, not sure where it's going). I was hoping to get a newer version than the EL7 build we made. It's working fine but there are some fixes and a GDPR feature to delete/anonymize accounts that are missing.

The other option is using epel9 but that's not there and even if Copr supports it partially getting all the deps in is gonna take years again (my experience with epel8).

Of course if we did a very good job in Fedora in time to get it in EL10 that would be great and future-proof but that's really going to be too late for what we already have to maintain.

If I was to be pragmatic I would use another distro I know well where the work has already been done and is working fine, with an Ansible role only needing minor changes, but I was told that was not the way to go…

Getting it working in EPEL9 is going to be significantly easier than EPEL8, since the default Python is 3.9. If we get stuff into Fedora, branching them into EPEL9-next and then EPEL9 final will be pretty straightforward.

I guess I could work on a specific Fedora release but working with rawhide is what I tried already and before I could complete anything a new major version of Python came in and once again unreleased patches needed to be backported, or new ones created and then you're not building on the version you're targeting anymore. 3.10 is already out, so how long do we have to complete just about everything before it becomes the default? How is this going to work?

Nowadays, Python versions are released yearly, so the next Python version will arrive for Fedora Linux 37. It's a lot less painful than the last time, when Python versions were coming irregularly and quickly.

Some servers were just restarted today. That shouldn't hurt no?

@ngompa I'm indeed facing problems (need new attrs, pytest, etc etc). I guess I can give a try to your proposal :-). I'm going to start again on our Copr space and I'll try to push what I can do. If you have any advice, I'm all ears.

I can try helping with postorius

I've created a tracker on Bugzilla for the packages we need:
https://bugzilla.redhat.com/show_bug.cgi?id=2030061 (alias: MailmanTracker)

python-hyperkitty was just packaged

I can try helping with postorius

postorius is here:
https://release-monitoring.org/project/54848/

added python-postorius fedora link above.

Looks like we are down to just 4 bugs on the tracker?

Perhaps we could make some kind of coordinated push to get it over the line now?

How best can we help?

The four bugs currently open are requests for co-maintainership. Is there anything we can do to move those forward?

Also, we need to package up python3-django3 in Fedora to fix postorius and hyperkitty FTIs in Fedora.

2030928 - done just waiting on ?
2031302 - appears done also?
2031342 - done, added @salimma as admin
2033064 - is django... a bit of doom there about 'must support for 10 years'

Last one?
https://bugzilla.redhat.com/show_bug.cgi?id=2033064

Update per chance?

@salimma was working on it and might have more detailed status but:

• mailman3 is in fedora, but fails to build currently ( https://bugzilla.redhat.com/show_bug.cgi?id=2113503 )
• hyperkitty is in fedora, but fails to build currently ( https://bugzilla.redhat.com/show_bug.cgi?id=2046878 )
• postorius is in fedora

rhel9 is blocked on django:
https://bugzilla.redhat.com/show_bug.cgi?id=2033064
but at this point someone could just do that.

Ideally we would get rhel9 done and then we could look at migrating to that.

mailman3 is deprecated. It is now maiman core: https://gitlab.com/mailman/mailman

Where would you like me to push https://gitlab.com/mailman/mailman ? Mailman3 or mailman in Anitya?

Fedora calls/called it 'mailman3' because the download/archive of the package is 'mailman' and we already had a mailman 2.x, so this became mailman3.

I don't understand your question on anitya... but here isn't the place to discuss that really. You should talk to the maintainers of the fedora/epel package, most likely by filing a bug in bugzilla.redhat.com?

This ticket is more for.. when there's a newer mailman available to move to, we will do that. :)

Anitya is already handled.

postorius fails to build currently: ( https://bugzilla.redhat.com/show_bug.cgi?id=2113645 )

FYI, there was an investigation done for CentOS mailing lists and their migration to mailman3, it could be helpful.

@salimma has already gone and fixed HyperKitty and Postorius and is working on Mailman3 itself now.

After this is sorted out, we'll start looking at branching packages for EPEL 9.

The infrastructure freeze is now over.

https://src.fedoraproject.org/rpms/python-flufl-bounce/
https://src.fedoraproject.org/rpms/python-flufl-i18n/
Need to be updated to the latest see: https://bugzilla.redhat.com/show_bug.cgi?id=2113503#c11

hyperkitty/mailman3 now updated to the latest
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/CBICRTHDEHRQQ6NOZPAWKZHFIFU7EOOE/

"Just waiting on the epel9 branch/build and we can look at upgrading.

kevin"
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/CBICRTHDEHRQQ6NOZPAWKZHFIFU7EOOE/

The last of Fedora's applicable downstream patches has finally landed in upstream Mailman!

https://gitlab.com/mailman/mailman/-/commit/a9b9a1a0388ee1c0b3308b698c6c18ce1af9bc76

hyperkitty update on the mailing list
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/YFLPI747U4LYCVWI7HVY46EJ54YYGNVK/

The last of the stack will be in EPEL 9 in the coming days, thanks to @salimma: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-4c6b819c9c

I will try to look at deploying the new mailman on staging next week.

Ack. @fab pointed out mailman web is missing, I'll try and get it packaged by then.

Cool.

Ack. @fab pointed out mailman web is missing, I'll try and get it packaged by then.

Any ETA? Thank you for your work.

@arrfab is working on a staging deployment now.

@arrfab is working on a staging deployment now.

I see, thanks.

Looking at what we have in Fedora staging deployment and we need to port https://src.fedoraproject.org/rpms/mailman3-fedmsg-plugin to EPEL 9 as well.

I was able to deploy mailman3-3.3.8-3.el9.noarch on staging https://lists.stg.fedoraproject.org/ it's not working correctly yet, but at least something is deployed and the URL shows it.

Everything is in ansible role. Only thing I needed to do manually was update of the DB mentioned in documentation

I was able to deploy mailman3-3.3.8-3.el9.noarch on staging https://lists.stg.fedoraproject.org/ it's not working correctly yet, but at least something is deployed and the URL shows it.

Everything is in ansible role. Only thing I needed to do manually was update of the DB mentioned in documentation

That is working now.

Some update about the migration.

1. mailman3 role is now in ansible repository. I hit a roadblock with current role and decided to start from scratch (the versions are so different I should did that from start)
2. The deployment is now using gunicorn and apache as reverse proxy (originally it was using WSGI and apache)
3. I found out that I need mailman3-fedmsg-plugin as archiver for our lists as the existing lists are complaining that it's missing

Currently I'm working on packaging the latest version of mailman3-fedmsg-plugin and it's dependencies in EPEL 9.

New updates:

• Staging instance is now using OIDC for authentication
• Creating and deleting lists works
• Sending verification e-mails works

What remains to be done:

• Test sending e-mail to a list
• Test redirecting staging OIDC authentication to production FAS - it was pointing to production FAS with OpenID and it's causing creation of new accounts in mailman with staging FAS. The redirection should help test if this will load the correct account
• Migrate prod to stg sync script
• Create systemd timers for tasks - they are now executed in parallel which is not ideal as it's causing OOM killer to kill some of the tasks

FYI, the CentOS mailing lists have switched over as of Monday: https://lists.centos.org/

New updates:

• Working on rewriting fedora auth provider for Django, which is part of django-mailman3 - allauth implementation of OIDC doesn't work well with fedoraproject.org aliases

What remains to be done:

• Test sending e-mail to a list
• Migrate prod to stg sync script
• Create systemd timers for tasks - they are now executed in parallel which is not ideal as it's causing OOM killer to kill some of the tasks

Good! Thanks for this. I appreciate it.

Just to document it here, for custom auth provider to work, you need to add it through /django-mailman interface as a new Social application. I need to find a way to automate this through ansible.