#5870 rawhide signing
Closed: Fixed 3 years ago Opened 6 years ago by kevin.

We have talked a number of times about getting rawhide packages signed, but haven't been able to come up with a solution that is secure and meets our needs. We should try and do so. :)

This came up again today because gnome-software has different code paths for signed/unsigned content and they would very much like rawhide to be signed so it tests the same code as for stable releases.

  • There is a koji plugin to sign all builds, but it's not implemented in a very nice way and stores it's keys/passphrases in clear text files on the hub.

  • Manually signing with sigul could be done, but since there's no gating, it would mean either large amounts of packages would go out unsigned or composes would fail for unsigned packages often.

  • Additional space would be taken up by more signed rpms/signatures.

  • Any solution we come up with could possibly be also used by copr, which also wishes to sign builds in an unattended manner.

Ideas welcome. ;)


Replying to [ticket:5870 kevin]:

We have talked a number of times about getting rawhide packages signed, but haven't been able to come up with a solution that is secure and meets our needs. We should try and do so. :)

To decide, whether it is secure, we need to decide which threats we want to protect against.

  • There is a koji plugin to sign all builds, but it's not implemented in a very nice way and stores it's keys/passphrases in clear text files on the hub.

What can be improved in the plugin besided the passphrase storage? There is no indication about improvements in the ticket I referenced, assuming it is the plugin you meant.

  • Additional space would be taken up by more signed rpms/signatures.

Are the signatures that big that this is a problem? Or is this related to the fact that the package might need to be re-signed when Rawhide is branched?

Storage wise its not teh signatures, but having two copies of rpms written to disk, one signed one unsigned.

I haven't really looked very deep at sigul before, but it is certainly possible to write a signing plugin. Are there barriers to sigul automation? What credentials would the hub need to have to accomplish signing via sigul?

So, there's (as always) a number of threats to consider.

Here's some of the things I don't like about that plugin:

Both the private key and passphrase are stored on the hub.

The hub is directly reachable by people on the net via it's web interface, so in theory a compromise of the hub could gain an attacker not only a way to sign whatever they want, but they could copy off the private key and passphrase and sign whatever they want somewhere else.

The private key and passphrase are accessable to anyone who otherwise has access to the machine (ie, an admin can copy off the private key and passphrase and use them somewhere else, or have them stolen from somewhere else).

Ideally the best way forward would be to work on sigul, and come up with a way to do non interactive signing. That would at least make it so the private keys are never exposed (they would only be on the sigul vault), and no one could take the key and sign stuff elsewhere. We could also run another system to do the signing requests and just have it watch builds and sign them as it goes... that would mean the hub itself wouldn't know the passphrase or how to sign things even.

For what is worth, I've been working on something similar for $dayjob:
https://github.com/network-box/kojihub-posttag-sign

It's a Koji Hub plugin which will sign packages when they are tagged. (the list of tags for which to sign packages is configurable).

You can configure a "signing command", which in our case is just a ssh call to our signing server (it takes the content of the rpm as stdin, and returns the signed rpm on stdout).

This way, neither the private key nor the passphrase are on the Koji Hub (they are only on our signing server, which is quite restricted).

Of course, that might not work for Fedora anyway. After all, Fedora has some additional constraints that we don't, most importantly that the Fedora machines are publicly accessible. (our stuff is all private to our internal network)

For copr, some folks are talking about:

https://github.com/openSUSE/obs-sign

It's written in c, pretty simple. It still has passphrases in files, but they can be on a remote machine that only allows some ip's to connect, etc. (It's not yet packaged that I know of for Fedora/EPEL either)

It would be nice to not have something and also sigul, but that might not be possible. ;(

So, the idea was floated to have a fedmsg listening service that signs packages.

I'm going to talk to Ralph about that some and also discuss with Mitr if we are going to get any sigul work this year and if he has any ideas on how to go about this...

So, we could easily make something that listens on fedmsg for builds and takes action on them. However, we need work to integrate that with sigul. We need someone(s) to work on that integration...

In the mean time to move this forward, perhaps we can do a 'best effort' version manually. Either:

a) Some packages not yet signed go out unsigned in rawhide composes.

b) we make rawhide build into f21-pending and only move them to f21 (and push out in compose) after signing.

a means some packages will simply not be signed at all.
b means some packages may be delayed getting out in composes if releng folks haven't signed them yet.

Replying to [comment:12 kevin]:

So, we could easily make something that listens on fedmsg for builds and takes action on them. However, we need work to integrate that with sigul. We need someone(s) to work on that integration...

In the mean time to move this forward, perhaps we can do a 'best effort' version manually. Either:

a) Some packages not yet signed go out unsigned in rawhide composes.

b) we make rawhide build into f21-pending and only move them to f21 (and push out in compose) after signing.

a means some packages will simply not be signed at all.
b means some packages may be delayed getting out in composes if releng folks haven't signed them yet.

So now there will be 2 repositories right?

The first one will be called ''Rawhide Signed'' and the second ''Rawhide Unsigned'', similar to how there is ''Updates'' and ''Updates Testing'' for F20?

Replying to [comment:13 quickbooks]:

So now there will be 2 repositories right?

The first one will be called ''Rawhide Signed'' and the second ''Rawhide Unsigned'', similar to how there is ''Updates'' and ''Updates Testing'' for F20?

there is currently no plans for changes, just looking at what we can do but the two options are that we try to sign all the packages but sometimes packages would go out unsigned.

or we add a gating process so only signed packages get out but sometimes there will be an additional delay in getting things pushed out to make sure that they are signed.

== Implementation idea ==

Create a proxy services that caches credentials for sigul and uses sigul to sign rawhide RPMs.

== Tasks ==
* Create new VM to host the service ([https://fedorahosted.org/fedora-infrastructure/ticket/4378 Infrastructure ticket 4378]): autosign01
* Allow till access to VM
* Setup VM to be able to run sigul
* Check whether koji has enough space to store signed RPMs (primary and secondary), implement removing of unneeded signed RPMs from kojipkgs if required
* Allow access to secondary sigul for autosign01
* Create sigul user for till primary and secondary Rawhide keys (f22)
* Make till koji admin for secondary kojis
* Add gating tag, e.g. f21-unsigned on primary koji and secondary kojis
* make all new builds go to the pending tag. After they are signed, they are moved to the final tag that is used for compose. Until the service is ready, manually tag builds into the staging tag to test it
* write service

== Open !Questions/Problems ==
* Sigul becomes unusable usually at least about once every two days and only dgilmore and kevin can restart it
* Add nagios(?) monitoring for sigul, e.g. check whether the TCP ports can be reached or add a monitoring key, that is not used for anything but the monitoring to check whether get-public-key works.
* Should we sign all builds for Rawhide, e.g. also builds on side-tags (is there a common pattern to identify these?)
* Which tag should be used? Maybe f21-updates-candidate can be renamed to f21-candidate and used for this
* Are there problems with garbage collection in koji, will koji remove signatures for Rawhide packages after 12 weeks?
* How to setup NTP, the current peers are not reachable by autosign, it is possible to sync against {{{66.187.233.4}}}
* Disable local fake clock in ntp (recommended by vmware for virtual machines, maybe it makes sense for KVM as well)
* autosign: postfix does not listen on IPv6 (ansible commit 16daf5eae8425aacfc4536e3818c0eb38cfe0c7d)
* How to monitor if services works?
* Add fedmsg message for signing events (sigul) or writing signed RPMs (koji)?

== Proxy Service Description ==
* Requires credentials for primary/secondary Rawhide keys before it is ready for operation, therefore e.g. ask on startup
* Uses credentials from individual users
* Store credentials in memory
* Listen on fedmsg
* Rawhide pacakges are build into a pending tag
* Whenenver a new Rawhide package is built on primary or secondary sign the respective RPMs using cached credentials
* After the package is signed, it is moved to a the tag rawhide is composed from

So, where are we here?

We have a autosign01 up and a signer on it.

Is it keeping up? If so, we could move on to looking at implementing the pending tag?

Replying to [comment:16 kevin]:

So, where are we here?

We have a autosign01 up and a signer on it.

Is it keeping up? If so, we could move on to looking at implementing the pending tag?

It is currently not running. I sign rawhide packages manually/slowly to make sure that not too many packages are unsigned. When it was running the last time, sigul was not stable enough which also caused problems for others who wanted to sign something. I am currently waiting for feedback on ticket:6023 to make the sigul outages less annoying.

taking off the meeting agenda until sigul is in better shape. looks like it may be with GSOC

We now have RoboSignatory doing non-gated rawhide signing.
As soon as a new fedpkg release with https://pagure.io/fedpkg/pull-request/48 is out, we should be able to enable gating autosigning on rawhide.

Metadata Update from @kevin:
- Issue assigned to puiterwijk
- Issue set to the milestone: Fedora 21 Final

3 years ago

rawhide is now signed

Metadata Update from @ausil:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

the fedora-repos-rawhide package for Fedora 24 and Fedora 25 still has gpgcheck=0 instead of gpgcheck=1 ... should a new, separate bug be created to address that issue?

the issue for this on https://bugzilla.redhat.com/show_bug.cgi?id=951841 points back to this issue

That is a whole different problem that has not yet been figured out

Login to comment on this ticket.

Metadata