#312 Enabling CI/CD: Experiment with building Atomic Host out of a module

Created 4 months ago by dustymabe
Modified 3 months ago

This started as an email to the atomic-devel list. Moving into a ticket for further discussion and tracking of experimentation of building Atomic Host out of a module. Summary of bullets from the email:

Here are some of the things we want to be able to do:

  • 1 Have a rigorous definition (including specific versions, buildroot) of
    what goes into an Atomic Host, including dependencies.

  • 2 Triggering the CI/CD pipeline based on a change to definition
    of what goes into Atomic Host.

  • 3 A way to revert a package in the Atomic Host compose to an earlier version.

  • 4 A place to store higher level tests along with rigorous definition
    of Atomic Host, including them being versioned.

  • 5 Landing multiple changes that need to land together to pass testing.

Out of that list we think we require that:

  • 1R The definition of what is composed into Atomic Host artifacts should
    include specific versions of packages, and all dependencies included.

  • 2R The definition of what is composed in an Atomic Host should be stored
    in a git repository so that changes can be detected easily. The
    CI/CD pipeline can be triggered off of changes to this reposiroty.

  • 3R A mechanism to make a future composed Atomic Host artifact, contain
    an earlier (in RPM NVR parlance) version of a package.

  • 4R The high level functional Atomic Host tests should live in the same
    git repository with the rigorous definition of what goes into an Atomic
    Host.

  • 5R A mechanism to tell the CI Pipeline that multiple dist-git repository
    changes (i.e. multiple changing RPMs) should be built and tested together.

Note - @walters also sent an email to a broader audience about modularity and Atomic Host that is worth a read.

@psabata, from modularity team, has volunteered to help us do a proof of concept on this

Edited 4 months ago by dustymabe
4 months ago

Metadata Update from @dustymabe:
- Issue tagged with: ci, host, modularity

4 months ago

Metadata Update from @dustymabe:
- Issue assigned to psabata

There's a lot of bugs to work through. I'm trying a build locally, and hitting a lot of variations of

https://github.com/cockpit-project/cockpit/issues/7510

But I'm a bit confused since some of the ones failing locally appear to be succeeding in koji; maybe newer RPM? One thing I haven't found yet is - the base-runtime etc. module is f26 only, is there a version of the host-and-platform somewhere?

Here's one that's currently very mysterious to me:
https://koji.fedoraproject.org/koji/buildinfo?buildID=955412
It looks like ostree's use of AC_USE_SYSTEM_EXTENSIONS is failing to define _GNU_SOURCE somehow.

There's other ones, like p11-kit fails with:

RPM build errors:
File not found: /builddir/build/BUILDROOT/p11-kit-0.23.7-3.module_55aa6181.x86_64/usr/share/man/man1/trust.1.gz

selinux-policy dies with:

libsepol.policydb_read: policydb version 31 does not match my version range 15-30

libxml2:

RPM build errors:
File not found: /builddir/build/BUILDROOT/libxml2-2.9.4-4.module_55aa6181.x86_64/usr/lib64/python3/site-packages/pycache/py*

etc.

Could you elaborate what exactly are you doing? Are you using mbs-build local for local tests?

The base-runtime module is indeed f26-only; it's being replaced by host and platform in f27. The atomic module depends on bootstrap:master for build, so that's the one mbs-build local will download for you -- beware, it's huge. Oh that and shim:master.

I can't comment on the build failures; I would need to dive into it some more. Sometimes mysterious things happen :/

I've updated some of the packages and made slight changes to the buildroot -- will kick off another build.

Should we track our findings here?

yes please

On another note, looking at the ostree rpm error I see:

error: Bad exit status from /var/tmp/rpm-tmp.Bvv1rE (%build)
    bogus date in %changelog: Sat Aug 25 2013 Colin Walters <walters@verbum.org> - 2013.5-3
    bogus date in %changelog: Sat Aug 25 2013 Colin Walters <walters@verbum.org> - 2013.5-2

are we building with a more strict version of the build tools in MBS vs normal koji?

are we building with a more strict version of the build tools in MBS vs normal koji?

Nope. These warnings are unrelated to the build failing and you have them in your passing builds, too.

Kicked off another build with @tdawson 's fixes. It will take a while to build & fail, of course. Feel free to watch the progress with mbs-build watch 806.

Ok, failures from the last build:

I need to leave early today and I'm traveling tomorrow so I'm not sure I can fix it all before the meeting but I'll try.

I have included fixes for (hopefully) everything listed above and kicked off another build. Watch the progress with mbs-build watch 812.

If it goes well, what's the next steps? Should I be pursuing new compose configuration in Fedora Infra or should OSTree images be created elsewhere?

atomic:master:20170822140830 built.

So...I am still uncertain about the distinct paths of:

1) Build fork of host&platform
2) Build on top of host&platform

The "fork" option carries with it a lot of maintenance overhead imposed by the modularity build system - we'll have distinct binaries as well, and that's going to invalidate some of the testing.

The "build on top" thing brings in a requirement to modularize all of the build dependencies. I know there's a prototype "autotools" module for example, but taking this out requires modules for a vast array of stuff, from golang out to corner cases like gtk-doc (currently requires perl), etc.

The "build on top" thing brings in a requirement to modularize all of the build dependencies. I know there's a prototype "autotools" module for example, but taking this out requires modules for a vast array of stuff, from golang out to corner cases like gtk-doc (currently requires perl), etc.

FWIW, I'm completely okay with a "atomic-build-deps" module; this can start as an ugly grab-bag of anything required, and can over time instead reference more specific other modules (and possibly eventually only reference other modules).

I feel like we should be able to do "build on top", possibly with "Atomic Host" as a host&platform Profile.

But atomic-build-deps would include huge amounts of stuff...just think about all of the individual golang BR packages. Just today that's already vast amounts of painful busywork that is ad-hoc automated via https://github.com/gofed/gofed - modularity means having another list of all of those packages. (This again is a general problem with modularity layering on top of all the stuff that exists now rather than fixing any of it)

Now at least for docker it looks like we gave up on trying to use split-out golang BRs. And hmm, skopeo also just uses with_bundled now too.

A couple of things here.

  • the "fork" option implies a lot of overhead indeed, however, most of it can be automated with simple scripts here and there -- automatic depsolving on dist-git changes, automatic rebuilds, that's all possible;
  • building on top of H&P is not only about having the build dependencies available, it also relies on currently undefined behavior where your module updates packages owned by another module (Atomic Host updating/overriding Host & Platform content in this case); we intend to make this impossible in the long term so I'm not sure how this would work
  • if Atomic Host is just a Platform profile, you cannot actually change anything in the sources -- no custom patches, minimized builds or alternative implementations; not even different versions of anything; furthermore, what is actually installed in the composed OSTree is defined in the JSON file, not the module (although I would be awesome if we could generate it from a profile later on)

Regarding atomic-build-deps, such a module would be more or less identical to bootstrap and I would discourage you from pursuing that path.

One thing that came up is we might be able to hack the "override" path via working around the fedora-release and fedora-modular-release package conflicts.

Login to comment on this ticket.