#418 Possibility for SIGs to create and publish DriverDiscs
Closed: Fixed with Explanation 2 years ago by arrfab. Opened 2 years ago by pjgeorg.

The kmods SIG is interested in creating and publishing DriverDiscs.

Creating a DriverDisc iso boils down to:

mkdir -p dd/rpms/$basearch
echo -e "Driver Update Disk version 3\c" > ./dd/rhdd3
pushd dd/rpms/$basearch

# Download rpms specified by a list

mkisofs -quiet -lR -V OEMDRV -input-charset utf8 -o kmods-sig-dd.iso ./dd

The kmods-sig-dd.iso should then be available on mirror.centos.org.

Is this something that can be done in CBS?

We prefer to do this in an automated way in CBS as we'll have to re-create it every time the kernel in the installer iso is updated. Afaik for CentOS Stream this means for every kernel release.

I read in the koji documentation, but all I could find there was about creating images using a kickstart file. However afaik one cannot create a DriverDisc using a kickstart file.

Metadata Update from @arrfab:
- Issue tagged with: feature-request

2 years ago

Metadata Update from @arrfab:
- Issue tagged with: cbs

2 years ago

koji doesn't seem to support that feature (see https://docs.pagure.org/koji/) but wondering if one can submit a RFE against pagure (so on https://pagure.io/koji/) to ask if that's something they'd plan. Do you want to take that one ? Once we have a way to do through koji (and that koji team will maintain) we can implement it. I guess it would be a new 'task"

Submitted a RFE, see https://pagure.io/koji/issue/2998

If you don't mind, I leave this issue open here to track record.

Metadata Update from @arrfab:
- Issue priority set to: Waiting on External (was: Needs Review)
- Issue tagged with: groomed

2 years ago

yes, it makes sense, so let's see if that's possible or if something else needs to be setup in parallel to build the same result...

@pjgeorg if Koji doesn't support that natively (still waiting for upstream answer on the ticket you opened) I have an idea about how to do that in parallel : either creating tags and modifying the taginfo and process to kick that somewhere else (where signing happens, through mqtt notifications), or even just have node subscribed to specific MQTT topic, restricted with ACLs (using ACO TLS certs, already in use for auth for koji) and that would parse the posted json payload and would build at the other end a DuD iso and push it (to testing or mirror).
Opinions ?

@arrfab Thanks for looking into this and commenting in upstream RFE. For me this is currently not a high priority issue (*). Hence I tend to suggest to wait a little longer and hope for some reaction concerning the upstream RFE. This simply seems to be the proper way of handling this use case.
Concerning the two suggested alternatives: I can live with both. From my perspective both are similar, hence it mainly depends on what is easier for you to implement (and maintain).

(*) For DuDs (and any of our kmods packages in general) to be really useful we'll probably need to be able to sign kernel modules first, see #307. Not sure about the status there.

Summary of yesterday's discussion in the kmods SIG meeting (logs) and some additional thoughts from my side:

The idea is to have a single kmod per .iso.
In most cases users probably only need one kmod during install. For all other use cases it is probably best to create a personalized Driver Disc anyway.

Having only a single kmod per .iso allows re-using the rpm NEVR for the iso name (and versioning if required).
My suggestion is to simply use <nevr>.<arch>.iso, i.e. the same name as the rpm with a simple /rpm/iso/

@arrfab mentioned yesterday that it might be possible to add another logic to tag-build. I.e. when you tag-build something to -testing or -release the Driver Disc is created and the artifact is pushed to buildlogs.c.o or mirror.c.o
In my opinion it would be sufficient to create DDs on tag-build to -release.

Open questions:

  • Path artifacts (DDs) are pushed to on mirror.c.o. Should the DDs be pushed to /images like other isos?
  • At some point we might add packages that are not kmods. Can the added logic be restricted to certain packages by some sort of filter?
    Possible filters are (i) only for packages with suffix kmod- or (ii) only for packages that do provide "kernel-modules" (following Driver Disc standard, see 2).

Please let me know what you think about this approach and what further information from my side is required.

Note: This is not a top priority on our side. I.e. in case it is too much work to establish such workflow we can live without it.
For now we can point users to a script (3) that makes it easy to create DDs with desired packages released by the kmods SIG.

Metadata Update from @arrfab:
- Issue assigned to arrfab

2 years ago

Having one .iso per NEVR/kmod would work I guess so we can do that.
I'm just wondering about stream installers which are always using different kernel version and so people would be confused about which .iso/DuD to use as it can work today but not tomorrow, etc.
So wondering if a generic kmod-<name>.iso that would have all possible NEVR for that kmod wouldn't be also a good thing to provide on regular basis.
Thoughts ?

WRT images, you meant under /kmods/<arch> ? (example would be http://mirror.centos.org/centos/8-stream/kmods/x86_64/)

@arrfab @pjgeorg I was spending some time building a Koji DUD plugin that is able to take a given rpm from the tag repos (and only from the tag repos), download it, and run the appropriate commands using mkisofs to then expose the produces ISO as a build result.

I was basing my plugin on https://pagure.io/koji/pull-request/3069 as an inspiration. I was planning on accepting any type of rpm. As you said you can prepare DUDs with not only kmods.

I think I will soon be able to show you a working PoC if you are interested.

@djuarezg as ticket https://pagure.io/koji/issue/2998 mentioned that it wasn't something koji would support anytime soon, we then thought we can do it outside of koji but still happening through koji as each tag-build operation is sending a notification that is caught on another machine that can just parse koji and build .iso
If you now come with the missing part in koji itself, that would be really awesome but ideally can you then comment on https://pagure.io/koji/issue/2998 and so we'd have something that would be eventually added into koji itself ?

WRT images, you meant under /kmods/<arch> ? (example would be http://mirror.centos.org/centos/8-stream/kmods/x86_64/)

I meant /kmods/<arch>/<project>-<release>
E.g. for rpms are currently under http://mirror.centos.org/centos/8-stream/kmods/x86_64/packages-main/Packages/k/, the DD images would then be found under http://mirror.centos.org/centos/8-stream/kmods/x86_64/packages-main/images (with or without the "/k" at the end, I do not mind).

I'm just wondering about stream installers which are always using different kernel version and so people would be confused about which .iso/DuD to use as it can work today but not tomorrow, etc.
So wondering if a generic kmod-<name>.iso that would have all possible NEVR for that kmod wouldn't be also a good thing to provide on regular basis.
Thoughts ?

Sounds like a good idea.
My understanding of how Anaconda deals with DDs with multiple versions of the same kmod is limited. Hence I dig a little bit in the documentation and its source code. Assuming the DD Readme is correct, having multiple NEVR for the same kmod might be an issue: All kmod rpms are extracted (not installed) and kernel modules are moved to /lib/modules/$(uname -r)/updates, see driver_updates.py#L55. This is required to pickup kabi tracking kmods (remember, packages are only extracted, not installed, i.e. weak-modules --add-modules is not called).
However this is an issue in case we have multiple NEVR of the same kmod. Before extracting a package only the kmod rpm's Provide kernel-modules is compared against the currently running kernel version. Having in mind that all kmod packages use >= for their Provide kernel-modules we'd easily end up with all NEVRs of a kmod package fulfilling this requirement. I.e. all NEVRs of the kmod package would be extracted and the kernel modules moved to /lib/modules/$(uname -r)/updates in an undetermined order and (probably) overwriting each other.

In addition all NEVRs that have been extracted will be added to a list of packages to be installed during installation. However some of the packages might not be installable due to unresolvable dependencies. I.e. an old NEVR of the kmod package that lists Provides kernel-modules >= x, but is not compatible with kernel version > x and we happen to run kernel version x+1.

Assuming my understanding is correct this could be easily fixed by proposing a change to dd_list or in driver_updates.py#L155 to only list the newest version of a package that fulfills the Provides kernel-modules requirement.

@arrfab @pjgeorg Sorry for the delay, in the response, holidays + illness.

@djuarezg as ticket https://pagure.io/koji/issue/2998 mentioned that it wasn't something koji would support anytime soon

I was in contact with @tkopecek and informed him of our willingness to help on this given that was indeed their position. I have an outdated (possibly non working as we speak until fixed) draft on https://gitlab.cern.ch/djuarezg/testing_dud_building though. I will try to get you a test version ASAP.

My knowledge on DUDs with multiple versions is limited but my initial draft assumes you would have a DUD per package, and regarding your use case, the DUD naming could be user-defined to help identifying which DUD covers which kernel version. The reason why I was only focusing on one rpm per ISO is that during installation you would always be able to include as many as you would want (unless there is a hard limit I have not found documented). As a reference, elrepo (https://elrepo.org/linux/dud/el8/x86_64/) uses this convention I was planning to follow.

Any thoughts?

@djuarezg One rpm per iso seems to be the usual approach. This definitely makes sense for RHEL as you can include el<major>_<minor> in the DUD iso name and there's only one installer iso for each minor release.

For CentOS Stream the installer iso is updated with every new compose. These composes are differentiated by including <date> and a <release> number in their name. There is no obvious way to get from CentOS Stream compose date to the required version of a kmod package - even if the kernel version is included in the DUD iso name.
A DUD containing various versions (compiled for different kernel versions) of a kmod package might be helpful in this case.
We could also solve this issue if we can somehow get the compose "id" (<date>.<release>) a particular kernel version was introduced first and add this to the DuD iso name. This would allow users to select the correct version by comparing the dates. However I fear this information is not available (in particular not in CBS).

I think we should definitely provide DUDs following the "one rpm per iso" approach. Anything on top is a nice to have.

Metadata Update from @arrfab:
- Assignee reset

2 years ago

@djuarezg hey .. was wondering if you had still a kind of PoC with upstream koji. Basically would like to know which direction we want to take to help the Kmods SIG : either have a native koji plugin/task, or just have a simpler bash wrapper that will be plumbed into sign+push process (that is triggered through tag-build operations at koji level)

TLDR; the PoC builds DUD ISOs, one rpm per ISO. Broken for tags with multiple archs, build one arch by one.

@arrfab https://gitlab.cern.ch/djuarezg/testing_dud_building still applies. I added my latest changes on the 20 Dec before leaving on holidays. I was stuck as there is a bug (also present in upstream's Kiwi plugin I took inspiration from) that does not allow to build DUD ISOs on tags with multiple archs.

Anyway, the proposed example only takes one rpm as a parameter, not multiple ones. Nonetheless I can add support for an arbitrary number of rpms which would be included in the ISO. Correct me if I am wrong but this, combined with the possibility to set the DUD ISO name with user input seems to cover what you mentioned on your last comment.

PS: Will submit the upstream PR when possible, could not do it any time before.

@pjgeorg , @djuarezg : so, from a CentOS Infra perspective, we have to decide which one to provide and then to maintain.

Ideally , if we want to go the koji route, it has to be available upstream (to ensure that it will continue to work) and then available as rpm pkg for easy consumption.

If we just plumb some script outside of koji, that means that it's not tied to any version, but would just be triggered by tag-build at koji level, and would build+push automatically some .iso.

Opinions and thoughts ?

From my ("user") perspective both approaches are fine. Essentially we want to be able to produce a DuD (one for every arch a package is built for) triggered by tag-build to release (we prefer to include signed rpms only) or by manual invocation. Anything else, e.g. multiple versions of a package in one DuD, are optional.

In case both approaches satisfy these requirements my opinion is simply whichever approach is less work for you to setup and maintain.

Feel free to comment on https://pagure.io/koji/pull-request/3217. Far from ideal yet, but ready to accept feedback.

[backlog refinement]
This is still a legit request, but we need to discuss the possibilities how to do this.

[backlog refinement]
We have a potential solution by responding to tag, but this ticket is currently not a priority.

[backlog refinement]
The koji issue related to this ticket was closed and we should wait for new version to be released and we should try to deploy it afterwards.

koji 1.29.0 should include the commit that fixes this issue

+1 , it seems it landed there but unfortunately, no real doc about how to use it.
As I'll be away for 3 weeks, either I'll be looking at this when I'm back, or someone from "Infra and releng" team can start looking at it.

plan :

  • have a public cbs.stg.centos.org env (minimal but good to test things publicly)
  • update it with koji 1.29.0
  • once validated (with ansible roles updated to reflect DUD plugin) we'll just replay on cbs.centos.org

@pjgeorg can't find any documentation about how to plumb that, so still wondering if the easiest option is to just plumb stupid functions in our existing sign+push process, and add some steps for .iso creation .. probably easier/faster than waiting for koji to have these features properly implemented and documented and working also for all architectures (it seems it wasn't even working in what was merged, based on some comments from @djuarezg )

I couldn't find any documentation except what is provided in https://pagure.io/koji/pull-request/3217 directly either. Both options work for me.
It is probably more of a question to yourself whether you want to maintain some additional functions in your existing sign+push process or wait for proper implementation and documentation in koji.

Metadata Update from @arrfab:
- Issue assigned to arrfab

2 years ago

it's easy to just plumb in our existing script so I started to work on it and have a PoC already in dev .. :

├── dud
│   └── kmod-hpsa-5.14.0~145-1.el9s.x86_64.iso
├── kojipkgs
├── Packages
│   └── k
│       └── kmod-hpsa-5.14.0~145-1.el9s.x86_64.rpm
├── pkglist
└── repodata
    ├── 0212d6398d09308ff51dd2f0ee59d3a535dff82423ce0aff9b599421b52e3ab8-filelists.sqlite.bz2
    ├── 0bb984182cc075365fdb40e1e6a33c1471551925e04407d1c3374d0117c1c934-primary.xml.gz
    ├── 9803e16001d0301fdb14593e2d42adff94f934db3ad4f5a8b422367f637a4fde-other.xml.gz
    ├── a43635a81a3f145747a31d76456e24161523e5b9d3c7d9e43808bb999ec2f764-other.sqlite.bz2
    ├── ae84076d124907811ad2b07c9ea86492478dca5010e91426628a2598d5e103fc-primary.sqlite.bz2
    ├── b4465aba4bfd58d701cc6518a124827082e75cab1e803622ae1ec499486f1e50-filelists.xml.gz
    └── repomd.xml

Just testing with single kmod in a testing tag on dev env to check but it seems to work.
@pjgeorg : dud directory is just a name I found but feel free to come with other name if you prefer

I quickly gave it a try on real cbs for the kmods9s-packages-rebuild-testing tag and it landed here :
https://buildlogs.centos.org/centos/9-stream/kmods/x86_64/packages-rebuild/dud/ (and other arches too)

Let me know if that works for you and if so, we can close this ticket :-)

Metadata Update from @arrfab:
- Issue untagged with: blocked
- Issue priority set to: Waiting on Reporter (was: Waiting on External)

2 years ago

Metadata Update from @arrfab:
- Issue tagged with: high-gain, medium-trouble

2 years ago

I checked one of the generated isos, all looks good to me. Thanks!
I also like that there is no manual work required, but instead a DriverDisc is automatically created for every kmod-* package tagged for -testing and -release.

Concerning the name of the directory :
dud if fine for me, but having in mind that the directory for the rpm packages is named Packages, i.e. no abbreviation, it'd be consistent to also not use an abbreviation for this directory. My personal preference is DriverDiscs following the Anaconda documentation 1.

Modified to use DriverDiscs instead and pushed the change in ansible.
For -testing it then landed on https://buildlogs.centos.org/centos/9-stream/kmods/x86_64/packages-rebuild/DriverDiscs/

Let me then close this ticket as it's seems to be doing what you initially requested for :)

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Boards 1
CBS Status: Backlog