No comment?

Just FYI, I am working on ascii_binder and it should be available in Fedora soonish ...

:100: :thumbsup: for the ability to send guidelines PRs.

I'm very much in favor of this.

I would like to put in some encouragement for AsciiBinder —I'd like https://docs.fedoraproject.org to be "one stop shopping" for user and contributor docs. But I'm generally happy with anything that moves us away from using the wiki for this kind of thing. (Let's use the wiki for what it's good at!) I know @bex had talked about possibly having a mechanism for publishing RST on that site as well as AsciiDoc, so maybe that's a possibility.

I note some irony in the formatting of the initial post here, where Markdown supports

# this

or this
=====

for headers but not

=== this ===

I also encourage you to consider AsciiDoc format and using AsciiBinder.

Today we have the ability for FPC to have it's own repo that it maintains that is published to docs.fp.o. This gets you "Version control, better formatting, local editing, pull requests, better indexing and searching." and doesn't make you get into the publishing business.

It will also put our packaging guide, a critical piece of Fedora documentation, in a visible location with other Fedora community documentation.

If you decide to move forward, please don't hesitate to ping me for details. If you'd like to do a PoC conversion, I am happy to help there too.

I tried to outline that I don't consider asciibinder inferior in my original post, and that sphinx is mostly personal preference / familiarity. I think it'd be ideal to do trial conversions in both formats and compare the outputs. This will be a bit of work, but might save us a lot of work in the future. In particular, I think interlinking and searching are something that is very important, but you can't really test without actually doing a part of the conversion. @bex, would you be willing to put up an asciibinder versions with at least a few pages?

Today we have the ability for FPC to have it's own repo that it maintains that is published to docs.fp.o.

Hopefully, we'll be able to serve the Guidelines from anywhere, after all once rendered, it's just a subtree of some html pages.

For the record I'm totally willing to be involved with something like this as long as:

• It isn't too terribly painful to write. I can deal with markdown and some rst but am not enthralled with having to learn yet another markup language. (Which I do realize is actually a thing.) But I can take the hit if it makes things easier by some measure. Mediawiki is pretty painful so something has to be pretty bad to not qualify as an improvement.

• It's not too difficult to get the end document generated. We're busy and sometimes a quick guidelines tweak will fit into the couple of minutes I have, while that plus a lengthy procedure to get a generated document live won't. I know that at least sphinx can take absolutely forever to generate things (witness the ansible documentation generation) and so I'm worried about any sphinx involvement.

• We don't lose history. (I know this was covered initially, but other markups might not have an automated conversion from a wiki.)

A secondary concern would be whether we can keep dynamic things like pre-collapsed sections (like the macros section of https://fedoraproject.org/wiki/Packaging:Python) or compact layouts like the side-pulled TOC blocks we use. Print-oriented markup systems rarely allow such things, but if the idea is to make the guidelines reasonably compact for those who have to read them. (And I know we lost a lot of compactness with the wiki retheme, which seems to be completely intentional for whatever reason. Fortunately I got it all back with a greasemonkey script.)

@tibbs

1. It is annoying that it's yet another markup, but I've found it to be a pretty easy one. It's more expressive than markdown, for whatever that's worth. It's definitely better than mediawiki.
2. The asciibinder thing takes seconds. Our plan — hopefully in place in a month or two at the latest — is that git commits go to https://docs.stg.fedoraproject.org/ immediately and then go live on the next hourly push to the main docs site.
3. Hmmm, several idea for history. What if we just stick the mediawiki markup in git (possibly replaying the history if that can be done automatically) and not convert each revision?

What @mattdm said ;) To expand a bit: both rst and asciidoc are rather straightforward. In my testing asciibinder seemed a bit faster, but we're talking seconds here either way. Sphinx supports (and asciibinder too, I think) partial regeneration, so when editing some document the regeneration cycle is nearly instantaneous.

The question of history is somewhat complicated. When the initial conversion to mw was done, previous history was not converted, so it seems lost. So below I'm only talking about the mediawiki era and the future.

Both @fale and I independently converted full history in mediawiki to git. We did some small things differently, so before making the final conversion it'd probably be useful to combine our scripts to get the best result. Nevertheless, in both conversions this history is for files in the mediawiki format. The conversion to asciidoc/rst is done as the last step resulting in one or more commits with the conversion and fixups. An alternative approach would be to do the mediawiki→new format conversion on each git commit. This would give nicer git blame/git log output. Unfortunately the results of automatic conversion are fairly bad, essentially because mw is so unclear and heavily mixes presentation and syntax, and a lot of manual fixups are required to actually get coherent output. So doing this for all commits is not feasible, and the approach with the history first in mw format and then the new format is the best option. But I still think such mixed history is quite useful. At least searching for phrases and such works and the origin of each piece of text can be determined with a bit of an effort.

What is needed from FPC is to declare that / if:
- you want to do this conversion
- which format you want to go with
- there are some features in the output that are absolutely required
We have POCs in the two proposed formats, and volunteers to do the full conversion, but nobody is going to put too much effort into this until FPC commits to something.

A secondary concern would be whether we can keep dynamic things like pre-collapsed sections (like the macros section of https://fedoraproject.org/wiki/Packaging:Python)

sphinx makes it very easy to add arbitrary .css. For something as simple as collapsible sections, I expect that just marking those blocks with an html class and adding a few lines of css will be enough. https://docs.python.org/3/ is a good showcase for sphinx.

[ It's also not hard to add arbitrary transformations to the output html or even generate sources on the fly. For example, a long time ago, I worked on sphinx docs to save all documentation examples embedded in the rst text as downloadable files, see http://mdp-toolkit.sourceforge.net/examples/gng/gng.html#gng. Doing this kind of thing is quite easy for anyone who knows basic python. This is not needed for something as simple as collapsible sections. But it could be useful if, for example, we wanted to copy and format current contents of redhat-rpm-config every time sources are generated. Just saying. ]

Somebody else will have to answer for asciidoc.

Anyway, I imagine the following workflow for FPC:
- you have a copy of the git repo locally
- FPC decides to do some change, and either you implement and commit the changes, or merge some PR
- run sphinx-build or asciibinder
- rsync to the target site
This will give you a chance to review the new stuff locally, and to send the announcement at the same time, and will be very fast.

An alternative approach of using some CI to build and upload a new version whenever the upstream repo is modified, a la readthedocs. This would work too.

Modifying this to reflect AsciiBinder + Docs Site likely workflow:

Anyway, I imagine the following workflow for FPC:
- you have a copy of the git repo locally
- FPC decides to do some change, and either you implement and commit the changes, or merge some PR

• Run asciibinder locally to build a test copy and review it.

• Commit the change to the repository (via PR or whatever FPC wants to use)

• The publication occurs as a result of the commit

• It appears on the docs.stg.fp.o site within about 5 minutes
• It appears on docs.fp.o within about 60 minutes

What @mattdm said ;) To expand a bit: both rst and asciidoc are rather straightforward. In my testing asciibinder seemed a bit faster, but we're talking seconds here either way. Sphinx supports (and asciibinder too, I think) partial regeneration, so when editing some document the regeneration cycle is nearly instantaneous.

This is true. Also with a docs.fp.o/AsciiBinder solution publishing is automated so you don't have to do anything but local testing. Local testing only builds the guidelines, not the entire docs site.

The question of history is somewhat complicated. When the initial conversion to mw was done, previous history was not converted, so it seems lost. So below I'm only talking about the mediawiki era and the future.

I encourage following @mattdm's idea and just commiting media wiki history. Then do a single conversion to AsciiDoc and follow from that.

Both @fale and I independently converted full history in mediawiki to git. We did some small things differently, so before making the final conversion it'd probably be useful to combine our scripts to get the best result. Nevertheless, in both conversions this history is for files in the mediawiki format. The conversion to asciidoc/rst is done as the last step resulting in one or more commits with the conversion and fixups. An alternative approach would be to do the mediawiki→new format conversion on each git commit. This would give nicer git blame/git log output. Unfortunately the results of automatic conversion are fairly bad, essentially because mw is so unclear and heavily mixes presentation and syntax, and a lot of manual fixups are required to actually get coherent output. So doing this for all commits is not feasible, and the approach with the history first in mw format and then the new format is the best option. But I still think such mixed history is quite useful. At least searching for phrases and such works and the origin of each piece of text can be determined with a bit of an effort.
What is needed from FPC is to declare that / if:
- you want to do this conversion
- which format you want to go with
- there are some features in the output that are absolutely required
We have POCs in the two proposed formats, and volunteers to do the full conversion, but nobody is going to put too much effort into this until FPC commits to something.

I'd like to work with whomever has the AsciiDoc version of the POC to get it published on stage so that we can see it in place. Who is this?

A secondary concern would be whether we can keep dynamic things like pre-collapsed sections (like the macros section of https://fedoraproject.org/wiki/Packaging:Python)

Modifying the CSS to do this should be possible. I don't personally have the CSS skills to do it, but I am sure I can work with someone and have it running quickly.

Adding the html classes is easy in AsciiDoc.

After talking this over in person with @bex during DevConf, we decided to go with asciibinder in order to integrate with the new docs website better. I'll work on the conversion to asciidoc and after the initial conversion is done post the link here. Later on this will be hooked up into the new docs pipeline (either straight from a PR or in the staging instance, depending on whether the generation of the website from contributor repositories is ready or not). Stay tuned.

So, the new docs site is up at https://docs.fedoraproject.org/ — what do we need to do to move forward?

@zbyszek @bex where can I find your latest work so I could move this forward?

@ignatenkobrain My work is here: https://pagure.io/fork/zbyszek/packaging-guidelines/tree/asciidoc

IIRC it is in-line with hte last work done by @zbyszek before we hit pagure issues (which I believe are now fixed).

Feel free to join the antora hack (cal invite sent) if you want to collaborate.

Go Go Asciidoc :D

Sorry for the delay.

I pushed my latest work to https://pagure.io/packaging-guidelines as master branch, and renamed @fale's master branch to adoc-v1.

The new master branch has a bunch of commits generated by automatic conversion from mediawiki, and then a bunch of cleanup commits on top. Those cleanup commits are necessary because the automatic conversion is full of formatting errors, broken links, etc. My plan was to redo the automatic conversion every once in a while to catch up with updates in the wiki, and rebase the cleanups on top of that. I think it'd be good to to this rebase now, because in the last three months the wiki changes quite a bit. I think it's best to keep rewriting the master branch to keep history clean until the conversion becomes official and that version becomes the canonical one.

I gave @ignatenkobrain rights to the repo.

One thing to consider is page naming: I removed "Packaging:" from page names and moved the pages under packaging/, e.g. packaging/Web_assets.adoc, packaging/Guidelines.adoc. I'm not entirely sure that this is right, maybe it's better to keep the "Packaging:" prefix because it makes it easier to refer to those pages (e.g. saying "Packaging:Guidelines" makes it obvious which page is meant, but "packaging/Guidelines" doesn't have this history in Fedora).

Another thing to check is the set of converted pages. I think we currently have everything in https://fedoraproject.org/wiki/Category:Packaging_guidelines, with the Japanese translations removed, but we probably should add the EPEL guidelines, but also do some cleanups of old files (they will be preserved in history, not lost.)

I did the rebase now, and renamed master→adoc-v2 and pushed the new version as master.

We should certainly not add the EPEL guidelines (at this time, at least). They aren't maintained by this committee, and moving them into git would effectively remove them from community access since there are no editing restrictions in place currently.

So any moves in that direction would need to be coordinated with the EPEL folks and should happen separately from any move of the main guidelines.

We should certainly not add the EPEL guidelines (at this time, at least). They aren't maintained by this committee, and moving them into git would effectively remove them from community access since there are no editing restrictions in place currently.
So any moves in that direction would need to be coordinated with the EPEL folks and should happen separately from any move of the main guidelines.

I think suggesting to them that a separate repo to get the same benefits would be helpful. If they need any advice, let me know.

edit:added: We can actually present the guidelines side by side by allow them to be maintained separately. WE can even get them into the same navigation tree if that is desired.

How can we move this forward?

We discussed this during Flock, and the plan is to make yet another attempt at this.

Actually the guidelines are quite big. I want to convert the FESCo pages, which is like two or three fairly simple pages instead first, to get a better feel for the process.

Great news! Let me know how can I help. Testing, contributing to code, whatever. (Feel free to ping me on IRC with details (once I'm there).)

Possibly worth noting that Infrastructure is moving their docs to https://docs.fedoraproject.org as well, so we definitely have momentum towards "docs all in one place".

Let's discuss it tomorrow.

It's probably worth noting that while the central docs have stayed with asciidoc (not restructured text), we've switched from asciibinder to antora for the site builder. That's hinted at in the thread here but I don't think anyone spelled it out. This isn't a big change from a content point of view, but does affect the directory structure.

We talked about this at todays meeting (https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2018-09-06/fpc.2018-09-06-16.00.txt):

• #727 convert guidelines to git and restructured text (geppetto,
  16:05:52)
• ACTION: convert latest versions of wiki packages to new format,
  publish it, make wiki read-only (+1:6, 0:0, -1:0) (geppetto,
  16:11:35)

@james a quick read of the minutes/log didn't result in me understanding which format was actually adopted. Can you share which one was selected?

I think we talked about using the same tools as other projects hosted on docs.fedoraproject.org -- which would be antora / asciidoc?

Note: If / once somebody starts the conversion, let us know to stop editing the wiki.

https://docs.stg.fedoraproject.org/en-US/packaging-guidelines/
https://docs.fedoraproject.org/en-US/packaging-guidelines/

Before I push it to our repo, do we want to preserve all history? or to some extent? What about authors?

So far I've got this repo: https://pagure.io/fork/ignatenkobrain/packaging-committee/branch/guidelines

History -> yes, with the "commit" messages. I don't think the authorship here is so important, all commits should be typo fixes or should reference an FPC issue, however it would be nicer.

The page looks good, but it seems like it doesn't match the latest state of the wiki page. (For example, there's no "No External Kernel Modules" section in the Packaging:Guidelines wiki page (anymore?).

Right, the last commit appears to be 2 years old.

The links in last section [1] should be updated. Actually, probably all the "Packaging:" references are not properly converted into appropriate hyperlinks.

Also, I am wondering if the guidelines should live in this repository? If the answer is YES, then they should live probably in some subdirectory?

Also, I am wondering if the guidelines should live in this repository? If the answer is YES, then they should live probably in some subdirectory?

I agree, I somehow assumed that they would end up in a packaging-guidelines repository or something (or even in a guidelines repository under a packaging namespace). Putting them in packaging-committee seems unexpected from the point of view of someone looking for the guidelines sources.

History -> yes, with the "commit" messages. I don't think the authorship here is so important, all commits should be typo fixes or should reference an FPC issue, however it would be nicer.

Exactly.

Re autohorship: in the past I tried to write an authors file (the mediawiki converter accepts a mediawiki login → git name and email mapping), but this doesn't work too well: some commits in mediawiki are done by "root" or contain bogus authorship, so it is not possible to assign authors to all changes. It looks messy if authorship for some commits is mapped and for other not. In the end, I think it's better to just leave the original mediawiki authorship. It is enough to identify who did which commit most of the time.

https://docs.stg.fedoraproject.org/en-US/packaging-guidelines/

@ignatenkobrain sorry, to say this, but now you have done the easy part ;]
Looking at the output document, it is obvious that most links are broken and/or are not even links, just plain text. In addition, formatting is broken in many places. https://docs.stg.fedoraproject.org/en-US/packaging-guidelines/Python/ is a pretty good example — it uses a lot of formatting, and in particular in the tables, it is quite a bit off. So it is necessary to go over each and every document and carefully fix formatting.

Take a look at commit in my latest conversion: https://pagure.io/packaging-guidelines/commits/master. I did a lot of cleanup there. Probably most of those commits still apply, and it would be nice not to repeat that work.

The question is how good the new version needs to be before it is accepted as the new official text. In my opinion, the look does not have to be perfect, and it's even OK if some links are broken, but the content, like all the rpm spec file snippets and macros are very important to be correct. People copy&paste from the guidelines into spec files, and having subtle (or not subtle) formatting errors would generate problems.

--

FYI, I now converted the FESCo wiki pages to antora and opened pull requests to have them integerated (see https://pagure.io/fesco/issue/1994#comment-531634 for the links to the repo and PRs). That is only a handful of pages (~5), and was much easier to do than the Guidelines, because I was able to go through all the text and fix all the links and formatting issues.

Based on the experience from the three conversions (initial mw→sphinx, and two mw→antora), my conclusion is that antora is not technologically ready for something as complicated as Packaging Guidelines.

I see the following issues:
- antora uses docker images to run the conversion and preview. This is slow, opaque, annoying. We really shouldn't require docker and root privileges for something as simple as converting a few text documents.
- link target titles are not propagated, and links must be titled manually. This means that when any section is retitled, it's necessary to go through all the refs and update them, or have them outdated. This may not sound too bad, but please try spending two hours going through documents and copying and pasting title of every feakin' link.
- file names leak into the content: links are written as filename.adoc[title]. This really shouldn't be necessary, and the name of the top-level anchor in the target document should be enough.
- formatting is hard to get right. For example, nested backquotes and stars have surprising results. For a document like the guidelines, where every character counts, this is particularly bad. When converting the guidelines I had cases where inserting a star-quoted word changed the appearance of nearby text. In another case I forgot to put a blank space underneath a title and the first paragraph was silently and completely omitted from output.
- there are no warning about broken links. Antora happily generates broken links to internal in-repo documents without any warning. Combined with the need to munge all links by hand, this is particularly bad.
- getting a preview of an individual repo is hard. If I want to modify docs-fp-o (the top-level repo), I can do this easily. But it is not possible to render one of the sub-repos (for example fesco-docs) properly on its own, because when it references other modules, and those links will be broken. So the only way to get a proper preview is to render the top-level repo and making the commits in the sub-repos visible to the top-level repo. This doesn't scale — the conversion will become slower and slower as we add more pages to docs.fp.o.

All in all, my conclusion is that FPC should seriously consider using sphinx for the Guideline documents. It avoids all the issues described above. I can see how antora has strengths for the Fedora website, with lots of content from many groups. FPC has a slowly-moving but very complex document managed by a small team of people, and the tradeoffs are completely different here.

On on authorship: I think a general "authors" file listing all names but not necessarily connecting individual bits to individual people is sufficient.

On sphinx vs. antora: note that antora is under very active development and responsive to our needs. We can get some of those things fixed. Additionally, getting the automatic CI pipeline actually functional should alleviate some of your concerns about previews (and broken links).

Also, try podman as non-root rather than docker. :)

Also, try podman as non-root rather than docker. :)

That's what my makefile does.

We discussed this ticket this week (https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2018-09-20/fpc.2018-09-20-16.01.txt):

• #727 convert guidelines to git and restructured text (geppetto,
  16:19:30)
• LINK:
  https://pagure.io/fork/ignatenkobrain/packaging-committee/commits/guidelines
  (ignatenkobrain__, 16:30:44)
• ACTION: Nobody should touch the wiki, at this point. (geppetto,
  16:46:25)

I've pushed my latest work in our repo.

https://docs.stg.fedoraproject.org/en-US/packaging-guidelines/
https://docs.fedoraproject.org/en-US/packaging-guidelines/

are re-generated once per day for now and will be done on-demand asap (on @asamalik).

You can generate docs locally using make (requires podman to be installed) and serve them using make serve.

I've fixed all links and markup on main page and on C/C++ one.

Everyone is now very welcomed to send pull requests :wine_glass:

@ignatenkobrain wow, this is looking really promising now.

If I have some spare time, I'll go through some of the other pages to check formatting and link validity :grin:

Should we keep track somewhere of which pages have already been checked for errors?

Out of curiosity, i've looked at:

https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/

Lacks any heading, says TOC at the top.

Some code examples show redundant pluses:

The macros table is totally wrong:

Is this supposed to be handfinished, or is it a bug in some conversion tool?

https://fedoraproject.org/wiki/Packaging_Committee#Guideline_Change_Procedure

This ^^ should be updated I guess as well as the README in the repo.

In Ruby guidelines, there are missing the "admon/warning" paragraphs. It is probably not the only place I suppose.

Also, it would be nice if the language specific guidelines are in a different section then the guidelines such as "AutoProvidesAndRequiresFiltering"

Is this supposed to be handfinished, or is it a bug in some conversion tool?

handfinished. Unfortunately conversion tool can't do everything for us.

In Ruby guidelines, there are missing the "admon/warning" paragraphs. It is probably not the only place I suppose.

I will look at this.

Also, it would be nice if the language specific guidelines are in a different section then the guidelines such as "AutoProvidesAndRequiresFiltering"

Yes, I agree. Feel free to send Pull Request (since we can do this now) ;)

Welcome to Fedora Docs! Glad to have the packaging guidelines there.

I've just pushed an update that lists the guidelines on the homepage.

@asamalik that's nice - however, it looks like it points to the wrong location. :wink:

@decathorpe I know, sorry, I've already pushed a fix.

OK, I have just found something completely disturbing. The antora software needed to process these documents is not actually in the distribution.

I absolutely cannot believe that nobody happened to mention this, so I'm feeling that I must simply not have neglected to read some piece of the discussion. Could someone point out where this was mentioned, because I'm not finding it in this ticket.

It doesn't help that the instructions for doing a local setup (which are just typing "make") don't work for me because I can't run podman:

ἐπιθυμία:~/work/packaging-committee/❯ make
podman run --rm -it -v "/home/tibbs/work/packaging-committee:/antora:z" \
        antora/antora --html-url-extension-style=indexify site.yml
error creating libpod runtime: exit status 1
make: *** [Makefile:5: doc] Error 125

I assume I need to grant myself additional permissions or maybe run as root. But I really shouldn't have to do complicated setup (or run things as root) just to turn markup into html. I should just be able to do dnf install antora and go, and then it would be up to me to use a container if I didn't want to drop those dependencies on my work system.

I feel that I really must have missed out on something, so I'm trying to not fly off the handle here. But we really shouldn't have made this switch until the software was at least available in updates-testing.

@tibbs Oh, I also assumed antora was available on fedora. I'm surprised it isn't, since I think all of https://docs.fedoraproject.org is built with it?

@ignatenkobrain Do you know what is needed to get antora packaged for fedora, or is there anything blocking it?

(BTW, installing podman and running make to build the pages worked for me, runing make serve did not, since the --directory switch it uses requires http.server from python 3.7.)

The antora software needed to process these documents is not actually in the distribution.
I absolutely cannot believe that nobody happened to mention this

First point in list of issues in https://pagure.io/packaging-committee/issue/727#comment-532031.

I'm +1 on the concerns here.

Also, I have 1 more concern. I believe that the conversion is not good enough to make people submit content PRs. I believe we should get the content readable before we move anything forward.

@decathorpe bunch of nodejs packages. I will provide better way of running webserver.

@churchyard it was expected that it won't be perfect. I fixed few pages myself and I will fix more as we go. It's just impossible to fix everything at once.

It doesn't help that the instructions for doing a local setup (which are just typing "make") don't work for me because I can't run podman:
I assume I need to grant myself additional permissions or maybe run as root.

Since I just figured this out when writing some patches, you need to enable rootless mode for podman to get it to work with the makefile.

(side note: pagure needs some less confusing formatting on blockquotes...)

FWIW, we could also use jekyll to build the pages (at least locally), as almost everything that's needed to do that is packaged for fedora already, only jekyll-asciidoc is missing, and I've submitted a review request for it today.

This all got done; there's really not much do discuss at this point.