#3 import systemd branch and build a package
Closed 3 years ago by dcavalca. Opened 3 years ago by dcavalca.

I think this is what we need based on https://wiki.centos.org/SIGGuide
- get a SIG branch on https://git.centos.org/rpms/systemd
- import https://github.com/facebookincubator/rpm-backports/tree/master/rpms/systemd there as a starting point
- request CBS tags (something like hyperscale8s-systemd-release and hyperscale8s-systemd-testing I guess?)
- the guide only covers how to do builds from a local src.rpm, but I think here we'd want to do them from git directly, especially for release builds
- to actually publish these to the mirrors, we'll also need a -release package and a SIG gpg key


@alphacc I'm a bit unclear on how to actually get a branch created -- do we file a ticket on centos-infra for that or is there an automated process? Also for building, I found https://pagure.io/centos-infra/issue/50 which seems relevant, but it's unclear what's the status there.

Reading the guide, it looks like the convention is to setup tags per-version, so we'd have things like hyperscale8s-systemd-246-release. I'm not sure though we want/need to treat every systemd version as its own repo (vs just having one repo for systemd with all the versions in it, under the assumption that one will always want the latest).

Also, probably a dumb question, but I didn't see this mentioned in the guide: once we publish packages, how do bug reports work? Specifically, can we get a dedicated version/category/etc on bugzilla for SIG packages? In the case of systemd, I'd expect users running our version would want to send bugs to us, not to the systemd CentOS maintainer.

As discussed:
- git builds are still somewhat experimental, but they're the way to go here and what we'll use once the documentation is updated
- tags == buildroot; in our case we can probably get by with just two for now (one for regular backports, one for experimental stuff), and have them feed into two repos

If the CBS Koji is new enough, we could have hyperscale SIG builds have packages set with a BugURL pointing to a location where we can take bug reports.

e.g. %bugurl https://pagure.io/centos-sig-hyperscale/packages or some such...

Ok, so in the end I think we want two repos, which I'm calling "stable" and "experimental", unless someone else has a better idea. "stable" would hold package backports (e.g. systemd) and modified rebuilds (e.g. iptables), and would have two tags:
- hyperscale8s-stable-release
- hyperscale8s-stable-testing

"experimental" would hold packages meant for testing or that could otherwise have unforseen side effects (e.g. the rpmcow stuff to being with):
- hyperscale8s-experimental-release
- hyperscale8s-experimental-testing

I'll go ahead and request these tags tomorrow unless someone objects.

@dcavalca what about

dist-git:

  • hyperscale8s
  • hyperscale8s-experimental

and in koji you will end up with:

  • hyperscale8s-build
  • hyperscale8s
  • hyperscale8s-testing

and same for -experimental..


I think it is easier for people to understand that it is not "stable" (as in RH stable).

UPD. and less to type :)

eh I'm all for keeping things shorter :) good call on "stable", it's definitely an overloaded term so best to avoid confusion. Why three tags though? I thought we just needed a "release" one and a "testing" one for each repo.

@dcavalca the -build is implicit, you have to have it to generate buildroot repo (and you can tag there buildtime-only packages, hopefully we won't do that). And release one I think easier to have without suffix.

@dcavalca one thing to consider before requesting experimental tags: shall those be used with or without non-experimental? That will influence how "release" package is set up and also how "tag inheritance" is set in Koji.

That's a good question. For rpmcow specifically, it's fairly independent from other stuff, so either would work. In general, I'd say it's probably reasonable to make experimental depend on the main repo to make things easier.

Ok so for future reference:
- ensure your SSH key in Pagure is good
- clone with git clone ssh://git@git.centos.org/rpms/systemd.git
- make a SIG branch with git checkout --orphan c8s-sig-hyperscale
- commit and push
- install https://copr.fedorainfracloud.org/coprs/bstinson/centos-packager/ and run centos-cert to get a new certificate if you don't have one already
- (I think) clone https://git.centos.org/centos-git-common and run lookaside_upload -f systemd-246.1.tar.gz -n systemd -b c8s-sig-hyperscale to upload the source tarball; this didn't actually work for me, though it did upload something:

$ ~/src/centos-git-common/lookaside_upload -f systemd-246.1.tar.gz -n systemd -b c8s-sig-hyperscale 
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : systemd-246.1.tar.gz 
[+] CentOS Lookaside upload tool -> Package name: systemd
[+] CentOS Lookaside upload tool -> sha1sum: 886c84cfa6861e1b367659b594722737d43d0fde
[+] CentOS Lookaside upload tool -> Remote branch: c8s-sig-hyperscale
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate                          

[+] CentOS Lookaside upload tool -> [ERROR] Something didn't work to push to https://git.centos.org/sources/systemd/c8s-sig-hyperscale/886c84cfa6861e1b367659b594722737d43d0fde
[+] CentOS Lookaside upload tool -> [ERROR] Verify at the server side
[dcavalca@tardis SOURCES]$ ~/src/centos-git-common/lookaside_upload -f systemd-246.1.tar.gz -n systemd -b c8s-sig-hyperscale 
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : systemd-246.1.tar.gz 
[+] CentOS Lookaside upload tool -> Package name: systemd
[+] CentOS Lookaside upload tool -> sha1sum: 886c84cfa6861e1b367659b594722737d43d0fde
[+] CentOS Lookaside upload tool -> Remote branch: c8s-sig-hyperscale
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

###################################################################################################### 100.0%curl: (22) The requested URL returned error: 403 Forbidden

[+] CentOS Lookaside upload tool -> [ERROR] Something didn't work to push to https://git.centos.org/sources/systemd/c8s-sig-hyperscale/886c84cfa6861e1b367659b594722737d43d0fde
[+] CentOS Lookaside upload tool -> [ERROR] Verify at the server side
$ 

Testing a scratch build with a local src.rpm in the meantime: https://cbs.centos.org/koji/taskinfo?taskID=1919143

I kicked this off with cbs build --scratch hyperscale8s-packages-main-el8 systemd-246.1-1.fb6.src.rpm

(this is just for testing, we'll change these to use hs or something else instead of fb for the suffix)

Probably .hs.el8 for our disttag

We should have our build environments have %dist .hs.el8 for hyperscale el8 targets.

@dcavalca ideally we'd be able to look together "in real time" but clearly it seems that cert isn't the correct one ?

curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate  

if your group membership on git.centos.org already shows sig-hyperscale, and that your ~/.centos.cert is correct, that's what is used to allow uploading to lookaside cache

Interesting, I'd completely missed that. I just tried again now and I don't get the SSL error anymore, but I do get a 403 at the end.

$ lookaside_upload -f systemd-246.1.tar.gz -n systemd -b c8s-sig-hyperscale
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : systemd-246.1.tar.gz 
[+] CentOS Lookaside upload tool -> Package name: systemd
[+] CentOS Lookaside upload tool -> sha1sum: 886c84cfa6861e1b367659b594722737d43d0fde
[+] CentOS Lookaside upload tool -> Remote branch: c8s-sig-hyperscale
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

###################################################################################################### 100.0%curl: (22) The requested URL returned error: 403 Forbidden

[+] CentOS Lookaside upload tool -> [ERROR] Something didn't work to push to https://git.centos.org/sources/systemd/c8s-sig-hyperscale/886c84cfa6861e1b367659b594722737d43d0fde
[+] CentOS Lookaside upload tool -> [ERROR] Verify at the server side
$ 

I think my cert is fine, but not sure how to actually check. I do know my membership on git.centos.org is fine, as I can push commits to the c8s-sig-hyperscale branch just fine.

@arrfab I can jump on IRC after the conference to poke at this if you'd like

@dcavalca I'll be in "family mode" directly after, but today myself I uploaded fine to lookaside, and it's showing my ACO username (from the CN: of the .centos.cert)
But I have some entries in log about people using wrong cert :

 AH02039: Certificate Verification: Error (18): self signed certificate

Does it verify ok against CA we use in ACO ?

cd ~
curl -O https://accounts.centos.org/ca/ca-cert.pem
openssl verify -CAfile ca-cert.pem .centos.cert

Yup that works:

$ openssl verify -CAfile ca-cert.pem .centos.cert
.centos.cert: OK

No worries, we can sort this out another time. Thanks for the help!

@dcavalca would be good testing again, as I had another look and no other SIG tried (so far) to upload to lookaside for c8s and realized that there is a regex used to filter good/bad branch names
The following commit should fix this : https://github.com/CentOS/ansible-role-pagure/commit/dbd3cff1437ac877310e0f8837a29e06851e9a01

Thanks @arrfab, I think that was it, this morning the upload worked just fine. Tried a git build in https://cbs.centos.org/koji/taskinfo?taskID=1927238 with cbs build --scratch hyperscale8s-packages-main-el8 git+https://git.centos.org/rpms/systemd.git#482fd4b94ecc46dc4cfa4b2f8f2ddbe11535bc31, looks like that failed with OSError: [Errno 2] No such file or directory: '/chroot_tmpdir/scmroot/systemd' during get_sources.

@dcavalca ideally we'd discuss that either on irc or centos-infra tracker, but here we go : the way the tags were created in the past , they had the mock.new_chroot setting set to 1 (so for nspawn), but during our tests we removed it from our testing tags .
I've now removed from all el8-build tags and that should now work for you (building from git)
I'd like to finalize announcement with @alphacc so that every SIG is aware that it now works

So just try to rebuild and it should work (as that's what I'm using for the infra tags)

And https://pagure.io/centos-infra/issue/212 for getting another tag for facebook builds as discussed during Dojo

Metadata Update from @dcavalca:
- Issue tagged with: systemd

3 years ago

Ok, everything looks good now:
- regular build: https://cbs.centos.org/koji/taskinfo?taskID=1934285
- fb build: https://cbs.centos.org/koji/taskinfo?taskID=1934288
Had to turn off audit for this to complete successfully, cut https://pagure.io/centos-sig-hyperscale/sig/issue/13 to track that. Going to close this now, and continue in https://pagure.io/centos-sig-hyperscale/sig/issue/4 with the mirror and release packages setup.

Metadata Update from @dcavalca:
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata