#5214 Setup koji tags and repo for Docs
Closed: Fixed None Opened 11 years ago by crantila.

Posted here as per [https://fedorahosted.org/fedora-infrastructure/ticket/3320]

'''phenomenon'''

To support simpler and more robust updates to docs.fp.o, need to setup koji to handle a new tag and repo for Docs.

'''implementation recommendation'''

<Sparks> What would be required to stand up a separate instance of koji for Docs? <dgilmore> Sparks: why would you want a seperate instance? <dgilmore> Sparks: i dont see any valid reason to do so <Sparks> dgilmore: For the Docs website. Publican has the ability publish documentation from packages (separate repo from the Fedora repo) for the website. We want to replace the git repo that operates it now. <dgilmore> Sparks: and why does that need a seperate koji? <Sparks> The git repo has gotten HUGE and is becoming an issue. <dgilmore> Sparks: so, why does that mean a seperate koji instance <dgilmore> Sparks: we could use a seperate tag and targets in koji <dgilmore> i dont see why it would need its own koji <Sparks> dgilmore: It's either that or try to get everyone setup as packagers. <dgilmore> Sparks: get everyone setup as packagers <Sparks> dgilmore: Except that they really won't be packagers. <dgilmore> Sparks: though you dont need to be a a packager to get a koji cert <dgilmore> you just need fas <dgilmore> Sparks: what would be the workflow? <Sparks> dgilmore: Okay, and with that we can send packages through koji and tag them separately? <Sparks> dgilmore: Basically we just tell Publican to build the package and submit it to koji. The Publican software does all the work. <dgilmore> Sparks: I still really dont know what your trying to do. pretend im an idiot(not really that hard) and explain what it is and how it should work <Sparks> dgilmore: I'm not far off... <Sparks> dgilmore: So Publican will make an SRPM package, submit it to koji destined for a repo. Our Publican backend will install those packages and publish the data to the website. <Sparks> ...as I understand it <dgilmore> Sparks: so we would need to set up seperate tags and tagets for it, defining a koji policy allowing srpms to be built. likely we would add a new group to koji and add people allowed to build docs to it and limit access to the tags/targets to people in that group <dgilmore> Sparks: so its all doable <Sparks> dgilmore: Well that's a lot easier. :) <Sparks> dgilmore: We already have a group in FAS for people that should have access (docs-publishers). Should I open a ticket? <dgilmore> Sparks: koji doesnt know anything about fas <dgilmore> Sparks: please file a ticket. We wont be able to make changes until after f17 is done <Sparks> dgilmore: That's fine. We're not completely ready for the transition so after F17 would be good. <Sparks> dgilmore: Thanks!


Noting that step 1 here is to have a dist-6E-epel-infra root in which we can build a more up-to-date Publican and the various deps that are too old in EL6 or EPEL.

We can't begin the packaging effort without this first step.

Dennis: Could we discuss this at FUDCon?

Dennis: We talked about this at FUDCon and you said that you had already completed part of this.

Has this been completed now?

Can you send details of the implementation, please?

Is there anything else needed from us?

I also said i needed you to ping me on irc so we can sort out the details and make sure it is done right which has not happened

Hi Dennis; to get started here, we will need at least these packages added to the new tag:

  • publican
  • wkhtmltopdf
  • wkhtmltopdf-qt
  • perl-DateTime-Format-DateParse
  • perl-HTML-Tree
  • perl-XML-TreeBuilder
  • perl-Locale-Maketext-Gettext
  • publican-fedora
  • Fedora_Documentation-web-home
  • Fedora-Burning_ISO_images_to_disc-18-web-en-US
  • Fedora-Fedora_Live_Images-18-web-en-US
  • Fedora-Installation_Guide-18-web-en-US
  • Fedora-Installation_Quick_Start_Guide-18-web-en-US
  • Fedora-Musicians_Guide-18-web-en-US
  • Fedora-Power_Management_Guide-18-web-en-US
  • Fedora-Release_Notes-18-web-en-US
  • Fedora-Security_Guide-18-web-en-US
  • Fedora-Technical_Notes-18-web-en-US
  • Fedora-UEFI_Secure_Boot_Guide-18-web-en-US
  • Fedora-Virtualization_Getting_Started_Guide-18-web-en-US
  • Fedora-Virtualization_Security_Guide-18-web-en-US

Hey Rudi,

Can you give us a high level overview of how this will work? As I understand it, docs publishers would take content from the docs git repos and herd it into koji to be built; a server on the other side of the process will have a script that updates the docs it publishes when the repo filled by our tag gets updated packages.

On the packages, is there a reason to follow the form of %{product}-%{name}-%{version}-%{format}-%{lang} for package names? To me, it would seem more logical to let the package name just be %{name}. It will get tedious and burdensome to task rel-eng with churning through package names for every language, document, and release we want to publish.

Thanks,
--Pete

Replying to [comment:6 immanetize]:

Can you give us a high level overview of how this will work? As I understand it, docs publishers would take content from the docs git repos and herd it into koji to be built; a server on the other side of the process will have a script that updates the docs it publishes when the repo filled by our tag gets updated packages.

Yes; that's precisely the mechanism. The only other considerations here are that the docs packages can be made available for folk to install on their own intranets if they want, or even on their own desktops (we build subpackages of the docs for local installations)

On the packages, is there a reason to follow the form of %{product}-%{name}-%{version}-%{format}-%{lang} for package names? To me, it would seem more logical to let the package name just be %{name}. It will get tedious and burdensome to task rel-eng with churning through package names for every language, document, and release we want to publish.

Long answer, because I acknlowledge that at first it looks like a lot of overkill :)

We're deliberately setting this up to be as flexible and extensible as possible; which means we need to consider potential namespace collisions for packagers, and for anyone who wants to install the documentation packages locally themselves. This is a very real risk when many of our %{name}s are as generic as "Installation Guide".

%{product} and %{version} pretty much ensure that a particular book title remains unambiguous.

We package versions and languages separately to ease maintenance. For example, a build problem in one language (because of a flaw in somebody's PO file) should never block somebody else releasing the book in their language. Also, a half-finished and unproofread translation should never get published and put on public display just because another translator finished their work long ago and is itching to release. And of course, we don't want to block the English release because not all translations are ready yet. Similar considerations apply to docs for specific Fedora versions. If a translation team only completed the translation of a particular book for F17, and has half-finished, broken translations for F18, F16, and F15, we need to be able to publish the F17 version of the book without also publishing broken translations for three other Fedora releases, plus empty translations (ie, English text with Translated headings provided by our various tools) for everything else all the way back to FC1.

%{format} doesn't require any extra repos -- we only build two formats, web and "desktop", and the "desktop" is a subpackage of -web-

Like I said, I know it looks like a lot of overkill at first, and I won't pretend that there isn't work for rel-eng to creating a bunch of new repos every Fedora release. Against that, the names of these repos are highly predictable and change only slowly from release to release; this task should be highly automatable. Also, the package naming structure implemented by Publican was designed around Red Hat's own practical experience in maintaining its own docs suite over the last ten years or so; the Publican team believes that it is "as simple as it can be, and no simpler" ;)

And a few more packages I've just realised we'll need in that buildroot and tag:

  • docbook-style-xsl
  • overpass-fonts
  • perl-File-Inplace
  • perl-HTML-FormatText-WithLinks
  • perl-HTML-FormatText-WithLinks-AndTables
  • perl-Sort-Versions
  • perl-String-Similarity
  • perl-Text-CSV_XS

I'm very confident that's it to get set up and running.

Hey Rudi, a few more questions, inline below.

Replying to [comment:7 rlandmann]:

Replying to [comment:6 immanetize]:

Can you give us a high level overview of how this will work? As I understand it, docs publishers would take content from the docs git repos and herd it into koji to be built; a server on the other side of the process will have a script that updates the docs it publishes when the repo filled by our tag gets updated packages.

Yes; that's precisely the mechanism. The only other considerations here are that the docs packages can be made available for folk to install on their own intranets if they want, or even on their own desktops (we build subpackages of the docs for local installations)

We've been discussing shipping docs in the branch repos as well, and doing so allows local utilities to parse them as well as the users. A -docs subpackage that must be downloaded from docs.fedoraproject.org and a document package from the f19 main repo seem redundant. RPMs only at docs.fp.o is more work for the users, who would have to get updates manually instead of inline with their other package updates.

On the packages, is there a reason to follow the form of %{product}-%{name}-%{version}-%{format}-%{lang} for package names? To me, it would seem more logical to let the package name just be %{name}. It will get tedious and burdensome to task rel-eng with churning through package names for every language, document, and release we want to publish.

Long answer, because I acknlowledge that at first it looks like a lot of overkill :)

We're deliberately setting this up to be as flexible and extensible as possible; which means we need to consider potential namespace collisions for packagers, and for anyone who wants to install the documentation packages locally themselves. This is a very real risk when many of our %{name}s are as generic as "Installation Guide".

%{product} and %{version} pretty much ensure that a particular book title remains unambiguous.

I facepalmed about half an hour after posting; of course, we have to have %{version} in the name, or we can't host multiple versions at once within the same tag.

We package versions and languages separately to ease maintenance. For example, a build problem in one language (because of a flaw in somebody's PO file) should never block somebody else releasing the book in their language. Also, a half-finished and unproofread translation should never get published and put on public display just because another translator finished their work long ago and is itching to release. And of course, we don't want to block the English release because not all translations are ready yet. Similar considerations apply to docs for specific Fedora versions. If a translation team only completed the translation of a particular book for F17, and has half-finished, broken translations for F18, F16, and F15, we need to be able to publish the F17 version of the book without also publishing broken translations for three other Fedora releases, plus empty translations (ie, English text with Translated headings provided by our various tools) for everything else all the way back to FC1.

I'm still missing something here. Versioned packages are logical, but language-specific ones not as much. If a translation has broken POs, we should be able to simply not include them. Given that we keep the release version branch as 'release ready,' for this release, we would simply limit translations committed in the origin/f19 branch to the ones that are sufficiently complete and that build. Broken languages only hold up the process for as long as it takes to identify them - which we'll have to do regardless. So, why not ship multi-lang RPMs?

%{format} doesn't require any extra repos -- we only build two formats, web and "desktop", and the "desktop" is a subpackage of -web-

Like I said, I know it looks like a lot of overkill at first, and I won't pretend that there isn't work for rel-eng to creating a bunch of new repos every Fedora release. Against that, the names of these repos are highly predictable and change only slowly from release to release; this task should be highly automatable. Also, the package naming structure implemented by Publican was designed around Red Hat's own practical experience in maintaining its own docs suite over the last ten years or so; the Publican team believes that it is "as simple as it can be, and no simpler" ;)

all the packages have been added. i need to still setup a policy allowing build from srpm

Replying to [comment:10 ausil]:

all the packages have been added. i need to still setup a policy allowing build from srpm

Thanks ausil! I see the el6-docs tag now :)

Should I separately request branches for those packages (and dist-git repos for those that don't have them) or will you set that up as part of this ticket?

Cheers and thanks again!

Rudi

Replying to [comment:12 rlandmann]:

Should I separately request branches for those packages (and dist-git repos for those that don't have them) or will you set that up as part of this ticket?

Actually, scratch that; after talking to AJ last week, I think that SRPM builds will be fine for everything that we need to do here.

Could you please go ahead and allow build_from_srpm and I think we're done here! :)

Cheers
Rudi

Is this ready? We still need to work on serving the content after Koji builds it...

Replying to [comment:14 immanetize]:

Is this ready? We still need to work on serving the content after Koji builds it...

True. This ticket only covers the Koji part of the work, which is independent of the rest to some degree.

All done! Many thanks ausil -- working perfectly now :)

Metadata Update from @crantila:
- Issue set to the milestone: Fedora 20 Beta
- Issue tagged with: docs

7 years ago

Login to comment on this ticket.

Metadata