#6119 AtomicHost rel-eng integration
Closed: Fixed None Opened 4 years ago by walters.

https://fedoraproject.org/wiki/Changes/AtomicHost currently uses rpm-ostree-toolbox:
https://github.com/projectatomic/rpm-ostree-toolbox/commits/

to generate:
- Installer
* Uses lorax
* But includes ostree content embedded
* Uses fedora-productimg-atomic
- Cloud images
* Uses ImageFactory and spin-kickstarts
- PXE-to-Live:
* Uses livemedia-creator

It runs everything (including ksflatten notably) inside automatically-updated Docker containers. This is more like what rel-eng tends to do with Mock, except Docker has a much simpler API for things like creating bind mounts and such.

There are a range of implementation strategies here for integration. We could run just the "installer" and "liveimage" -toolbox tasks on a compose box, but continue using Koji for the cloud images as we are today.

(-toolbox does cloud images via ImageFactory locally so I can conveniently test changes before uploading them)


  • [pxetolive] Dennis says that livemedia-creator must be run as a Koji task
  • [installer] Dennis says that the installer must be integrated into pungi for F23, but we might be able to do something short term for F22?
  • [other] Dennis says that rpm-ostree compose tree should become a Koji task

With the following:

https://github.com/rhinstaller/lorax/pull/8
https://git.fedorahosted.org/cgit/spin-kickstarts.git/commit/?id=0840fc1394830a3f970239f06cae4a2ef58ed85ft

This invocation should work to generate a functional Atomic F22 installer (substitute appropriate HTTP locations):

{{{
lorax --nomacboot -p 'Fedora Atomic' -v 22 -r 22 --source=http://127.0.0.1/repomirror/fedora-22 --add-template /srv/fedora-atomic/spin-kickstarts/atomic-installer/lorax-configure-repo.tmpl --add-template-var=ostree_osname=fedora-atomic --add-arch-template-var=ostree_repo=http://127.0.0.1/fedora-atomic/repo --add-template-var=ostree_ref=fedora-atomic/f22/x86_64/docker-host --add-arch-template /srv/fedora-atomic/spin-kickstarts/atomic-installer/lorax-embed-repo.tmpl --add-arch-template-var=ostree_osname=fedora-atomic --add-arch-template-var=ostree_ref=fedora-atomic/f22/x86_64/docker-host -i fedora-productimg-atomic /var/tmp/lorax
}}}

I think the best bet really is to have pungi make the iso. lorax really is not the right tool.

Pungi is a lot of complex code for injecting yum repositories, handling media splitting, dealing with multilib heuristics, etc.

None of this applies for atomic - there is no code to be shared. All the code that can be shared is already shared in lorax. The fact that the Atomic installer code is just a single (slightly verbose) shell script invocation on top of lorax is testament to that, right?

The above doesn't mean I'm fundamentally opposed to having pungi contain the above one line shell script - but please include rationales. "lorax really is not the right tool, <because reasons="">".

We could try to do this in rawhide first too and not block on the F22 freeze.

Also, so far we've just talked about the installer, but I'd like to hear what issues there are around rpm-ostree-toolbox -> livemedia-creator. Was the thought here that any deliverable has to come out of Koji, so it's having a Koji plugin for it? Basically just getting logging?

Or were there other things like wanting an ICICLE like verification that packages in the kickstart came from the tree?

Sorry I have been busy with other things, lorax is a tool that makes an anaconda install environment, and nothing else. it doesnt do things like getting content together, or putting together the content with the installer. That is how it was designed to work. longer term we should proably break out things like like the logic to make isos into its own library and share it between pungi and lorax. Doing it lorax is quite clunky, we then need to go in after and do extra steps.

I am working on invoking lorax in the compose scripts.

We want to do all the tasks in koji, we need to move making the atomic tree into koji, this is so that we can scale out the tasks and to provide a central location to get logs and to increase the transparency of the processes.I know people want atomic on arm, there seems to be no reason we should not support 32 bit x86 as well. Doing the work in koji lets us do things in parallel and much more simply run the tasks on the necessary arches. We do need to get it done in rawhide, realistically this should have gone into rawhide 6 months ago. Now is a great time for Rawhide for Fedora 23.

We do want to ensure that the packages are from koji, enable cross checking that we have the right bits.

I definitely understand the rationale behind pungi for mainline "installer + RPMs". But for Atomic, the content is by nature already assembled, and the point of the scripts is to inject it into the installer itself so it can be installed offline.

Now, I have had requests to provide the exact ostree commit that went into an installer image, and it makes a lot of sense to have visible. If we're trying to achieve that there are the exact same package versions in the release content set, I could certainly see an argument that "pungi" would generate two installers and for the ostree one, start from the exact commit used for that compose, rather than "latest".

On the other hand, you're talking about moving things into Koji with the argument that they can be asynchronous - but that is at odds with the concept of a gigantic release all containing one unified package set.

At a high level, the way I'm thinking about things, particularly with discussion of the 2-4-6 for Atomic:
- It's really important to share as much code as possible (hence using Anaconda, lorax, obviously RPMs, etc.). I'm not opposed to Koji as well (though there's more to discuss there, such as the fact that koji today has no concept of dependencies between tasks, e.g. rpm -> tree -> [cloud image | pxetolive]).
- On the other hand, one of the facets some competitors have is releasing a lot more frequently than every 6 months. And matching this is definitely part of the goal of Atomic - releasing the host tree as well as the base image shouldn't (need to) block on GPG signing texlive. (On the other other hand, some things included in Atomic have documentation built using TeX, so I know it's not that simple - but there's a reachable middle ground)

Anyways so...bottom line then I guess is for Fedora 22 there will be an Atomic-specific installer, but not PXE-to-Live? And the latter is blocked on a Koji task?

It's been a week, how does the installer look? Is there anything I can change in the code or configuration to make this easier?

And for PXE-to-Live, what needs to be done?

Another issue actually with PXE-to-Live that I should have raised earlier is that it cannot be updated online incrementally (via yum/rpm-ostree) - the point is you boot into the latest system. This necessitates a policy for respinning it frequently.

Similar discussions have been occuring on the Fedora Cloud side for image update cadence.

I have committed code to make the installer image as part of the compose process. for Fedora 23 it needs to get integrated into pungi. we will be using pungi 4 which is a big change from the pungi we have used to date.

As discussed in the FESCo meeting the pxe to live image is to be defered to fedora 23. we need to figure out how to build the product and what it will look like and how it will be consumed by end users. QA needs to be engaged as it will be a massive undertaking to QA it constantly.

removing the meeting keyword, but leaving open so we can discuss the changes that will need to happen for Fedora 23

moving to f23 for the work to get things properly integrated

Metadata Update from @walters:
- Issue set to the milestone: Fedora 23 Alpha

2 years ago

Login to comment on this ticket.

Metadata