#1002 Enable syndication of fedora mailing lists
Closed: Upstream 4 years ago Opened 12 years ago by nim.

We have many low-traffic lists such as fedora-announce where we publish information we really want spread far and wide.

Nowadays however most people do not bother subscribing to mailing lists and rely on web tools such as planets and other kinds of RSS/Atom agregators.

It is not realistic to expect Fedora contributors to first work on a feature, then announce in on a fedora announce list, then publish it again on a blog, then make the effort to have it syndicated in the right places. This workflow needs to be simplified to reduce the manual overhead if we do not want a lot of information to be lost and a lot of effort to be wasted instead of working on new contributions.

Lost information reduces Fedora's web presence. Less presence means less news articles and reviews on Fedora, less credits to Fedora contributors, less users and ultimately less contributions. Web presence is a major Fedora weakness compared to Ubuntu for example.

Fixing this only requires an RSS or Atom feed of our lists that could then be syndicated on Fedora planet and other special-purpose planets.

There are several ways to do it:

  1. replace mailman with new written-from-scratch software

  2. deploy one of the numerous mailman-to-rss mailman addons available on the web. Unfortunately
    - none of them seem blessed upstream,
    - googling does not reveal one which is obviously better than the others
    - this requires cooperation from the entity managing Fedora lists

  3. have those lists archived gmane-side, and use the RSS feeds of the gmane blog interface.
    - this makes Fedora rely for its communication on an external entity
    - gmane lists Fedora under Red Hat for historic reasons, which Red Hat does not want
    - this adds one more step to the creation of a fedora list, which will be forgotten most of the time

  4. have those lists transformed into blogs. BLogger for example allow blog content submission by e-mail (magic secret address to protect from spammers), and push of the resulting html files to a third-party (fedora-controled) web site. The rss/atom feed associated to the blog can then be syndicated.
    - this makes Fedora rely for its communication on an external entity
    - this requires setting up a mail address filter between mailman and blogger to avoid exposing the adresses in the list mails to spammers. For availability reasons this filter must be deployed on Fedora servers. Post-processing blogger files fedora-side after their push won't work because we can not control the time of the blogger push and blogger does not include a trigger system
    - this requires setting up ftp of sftp space fedora-side blogger can push to, and exposing this space in http to the web

  5. ???

Hey, sorry we haven't updated this in a while. I think for a while, we actually had fedora-announce-list setup to go to planet through an RSS feed (CCing kushal to see if he knows what ended up happening with that).

Are you sure that option 2 requires cooperation with the people that run redhat.com mailing lists? I recall that for the fedora-announce-list feed, all that needed to happen was subscribing a special email account to the list.

Just a few more thoughts:

The next major upstream version of mailman will have rss functionality built in. That doesn't really help us now though unless we want to backport the patch somehow:

One solution: https://support.mayfirst.org/browser/trunk/mailman2rss
(PHP and scrapes the archives).

Another: http://jmason.org/software/scripts/mailman-archive-to-rss.txt
(Perl and scrapes the archives)

Another: http://github.com/philgyford/mailman-archive-scraper/tree/master
(Python, copies all the archives and processes them).

If we could do a 'subscribe a rss user and have them process the rss' that might be cleaner than any archive scraping.

There are RSS feeds in gmane:

{{{RSS feeds:

  1. All messages from the list, with excerpted texts.
  2. Topics from the list, with excerpted texts.
  3. All messages from the list, with complete texts.
  4. Topics from the list, with complete texts.

Replying to [comment:5 till]:

I don't think it is a good idea to relay for such basic functionnality on a third party site, esp. one that mangles our list names in such a way

RSS gmane feeds were known when this ticket was submitted two years ago.

Per Matt Domsch's response to a discussion on the Infrastructure mailing list, I've got mailman-archive-to-rss.pl working from http://jmason.org/software/scripts/mailman-archive-to-rss.txt

It's a pretty slick script, the only things we need to figure out are these:

1) Which lists we want to syndicate and how to manage the list.
2) A test server with at least a smattering of mailing list archives so that I may test the script, a cronjob and setup the httpd config rules.


I have now completed the archive syndication for devel@, users@ and announce@. These rss can be accessed by visiting http://lists.fedoraproject.org/rss/<listname>.xml. For the announce@ list, point a browser at http://lists.fedoraproject.org/rss/announce.xml.

The cron job currently runs once an hour, if we decide we want it to be more often, please let me know. If there are any other requests, please also let me know.


Would it be possible to process the fonts and fonts-bugs lists too please?

Huh, just rediscovered this request, almost didn't even recognize it, I had worked on it so long ago. Is this still needed? I would probably still be able to perform the task.


How about we just do them all? have the script detect all valid lists and just generate rss for them all?

I wonder where the cron job for this is running. I can see it is actually working on a month-by-month basis.

Seems like somewhere on lists.fp.o would be the most likely candidate.

Kevin, is there a specific spot where cron jobs live? I could look there and update the @LISTS to include all the lists.

It's in modules/mailman/files/mailman-archive-to-rss.pl in puppet.

Ideally it would use list_lists or something to just run against them all ? (so we don't have to manually add lists to it as they are added).

Well, we have switched over to mailman3/hyperkitty since this was last updated, and the setup before isn't going to work with mailman3/hyperkitty.

I've filed https://gitlab.com/mailman/hyperkitty/issues/112 upstream against hyperkitty to try and get this properly implemented upstream. That way once we have it it will stay implemented.

I am now going to close this ticket as 'upstream' because the work really needs to happen there and having this ticket sit here does no particular good.

@kevin changed the status to Closed

4 years ago

Login to comment on this ticket.