#4857 private builds in koji
Closed: Fixed None Opened 8 years ago by kalev.

nirik and mclasen and I had a discussion at Flock how to best build an rpm privately. The use case for this would be the OpenH264 codec.

One idea was to have a .htaccess type of access control for disabling downloads for that particular package in koji - nirik said he'd investigate if it's feasible.


Ping, it would be great to have some progress on this. If we get this sorted out, it will provide tangible benefits to our users (easy access to legal codecs).

yes, this is on my todo list. I simply haven't gotten to it yet.

Please be patient and we will see if I can get something working, and if not consult with koji developers.

ok. I finally was able to look into this.

Here's how I think it could work:

For packages in this category we add to kojipkgs and koji httpd configs:

{{{
RedirectMatch "./this-package-name..x86_64.rpm$" "https://fedoraproject.org/wiki/non-distributable-rpms"
RedirectMatch "./this-package-name..armv7.rpm$" "https://fedoraproject.org/wiki/non-distributable-rpms"
RedirectMatch "./this-package-name..i386.rpm$" "https://fedoraproject.org/wiki/non-distributable-rpms"
}}}

(and similar for any secondary arches too in their kojis).

This would allow koji to build the package as normal, and have the src.rpm and info about the binary rpm and us to sign it even, but disallow people downloading the binary rpms.

Nothing would be able to BuildRequire the package, since it wouldn't be available to builders either. I assume that since its a plugin its a leaf node right?

We would need to figure out tagging. I guess it would have to be in it's own seperate tags so it never got pulled into a compose.

Is there the desire to utilize this mechanism to build embargoed security updates as well? If so, then you would also need to mask out the SRPM itself until the embargo is lifted. If this isn't intended to cover that, then please ignore (or maybe think about it separately).

no. I was thinking that as out of scope of this.

Embargoed builds would also need things like supressing fedmsgs and not showing the builds on the web interface as they were happening and other things with dist-git, etc.

All good points. You have clearly thought about it more than I have.

Thanks for looking at it, Kevin! Sounds like this should work.

Nothing would be able to BuildRequire the package, since it wouldn't be available to builders either. I assume that since its a plugin its a leaf node right?

Yup, I don't think it would be a problem.

Replying to [comment:4 kevin]:

ok. I finally was able to look into this.

Here's how I think it could work:

For packages in this category we add to kojipkgs and koji httpd configs:

{{{
RedirectMatch "./this-package-name..x86_64.rpm$" "https://fedoraproject.org/wiki/non-distributable-rpms"
RedirectMatch "./this-package-name..armv7.rpm$" "https://fedoraproject.org/wiki/non-distributable-rpms"
RedirectMatch "./this-package-name..i386.rpm$" "https://fedoraproject.org/wiki/non-distributable-rpms"
}}}

(and similar for any secondary arches too in their kojis).

This would allow koji to build the package as normal, and have the src.rpm and info about the binary rpm and us to sign it even, but disallow people downloading the binary rpms.

Nothing would be able to BuildRequire the package, since it wouldn't be available to builders either. I assume that since its a plugin its a leaf node right?

We would need to figure out tagging. I guess it would have to be in it's own seperate tags so it never got pulled into a compose.

you can still download the builds using koji's xmlrpc interface, which is how "koji download-build" works. I also have questions over how delivery to users would work. What mechanism would be in place to verify that the build is what users get. How would things be signed and how would it get onto the users systems. I also would not make any change here without the okay of Legal as we could be in a very sticky situation, especially if there is a critical security update that we can not get to users at all due to us not signing or having any control over delivery.

I don't know how to deal with the xmlrpc side. ;(

I think delivery would be:

  • Package is reviewed / added to fedora
  • Package is built
  • releng signs package and sends it to cisco
  • end users pull from cisco and can go to kojiweb and see the checksums of the package and that it's signed by us.

I agree we need to consider delivery times, etc...

I also think legal and fesco should sign off on anything we do here. ;)

I'm told that this has signoff from Richard Fontana.

It may be worth emphasizing that we are talking about building one particular package privately here. We are not asking for a generalized mechanism to do this for any package.

Replying to [comment:11 mattdm]:

I'm told that this has signoff from Richard Fontana.

Correct. The XMLRPC interface isn't something the general public would be expected to know or use, so the rewrites should be sufficient here. I believe the Legal box has been checked, but we still need to make sure FESCo knows the solution they contemplated in [https://fedorahosted.org/fesco/ticket/1359 their earlier ticket] is now available. From reading the meeting log, it sounds like this is the type of solution they were looking for (i.e. we can verify the build from source for users, but are not distributing the package).

Replying to [comment:12 mclasen]:

It may be worth emphasizing that we are talking about building one particular package privately here. We are not asking for a generalized mechanism to do this for any package.

Historically, almost every "we are talking about building one particular" ends up being the precedent for using the mechanism/path/tool/etc for generalized builds. It then falls about horribly at some point and everyone says "We shouldn't engineer 1 offs in the future" Then when people work on making the next special case not a 1-off it gets brought up that it is too slow and could we do this as a 1 time. Rinse lather repeat.

Replying to [comment:13 pfrields]:

Replying to [comment:11 mattdm]:

I'm told that this has signoff from Richard Fontana.

Correct. The XMLRPC interface isn't something the general public would be expected to know or use, so the rewrites should be sufficient here. I believe the Legal box has been checked, but we still need to make sure FESCo knows the solution they contemplated in [https://fedorahosted.org/fesco/ticket/1359 their earlier ticket] is now available. From reading the meeting log, it sounds like this is the type of solution they were looking for (i.e. we can verify the build from source for users, but are not distributing the package).

I guess my point about "koji download-build" using the xmlrpc interface was missed or ignored. users do not need to know anything about it all they need to know is that they can trivially get the rpms by executing "koji download-build openh264-1-1"

Replying to [comment:10 kevin]:

I don't know how to deal with the xmlrpc side. ;(

I think delivery would be:

  • Package is reviewed / added to fedora
  • Package is built
  • releng signs package and sends it to cisco
  • end users pull from cisco and can go to kojiweb and see the checksums of the package and that it's signed by us.

how do they get it from cisco? do we ship a .repo file that points them at the right location? what is the interface we have with cisco?

I agree we need to consider delivery times, etc...

I also think legal and fesco should sign off on anything we do here. ;)

we have no way to exclude the build from composes without writeing code, so the only other option will be making changes to tagging. It is probably going to need koji-gc changes and a special workflow. It is all things that have not been looked at on the releng side. Can someone please file an issue at http://taiga.cloud.fedoraproject.org/project/acarter-fedora-docker-atomic-tooling/ so that we can scaope out the work that would be needed and get it into the work queue.

Replying to [comment:16 ausil]:

how do they get it from cisco? do we ship a .repo file that points them at the right location? what is the interface we have with cisco?

The idea was that we'd ship .repo files pointing users to Cisco's servers. Everything would be signed by our regular keys, so we don't need anything else for verification. If it checks out against our own signing keys, we can trust the build; otherwise package managers would reject it.

One place where we could put the .repo file might be http://pkgs.fedoraproject.org/cgit/fedora-workstation-repositories.git/

Right now our interface with Cisco is an email interface, where we'd email them a tarball to upload to their servers. The tarball would contain everything we need; I guess we'd send them 4 repositories/directories for the last releases (F21-F24).

we have no way to exclude the build from composes without writeing code

If we don't tag builds with the special tags (f21, f21-updates, f21-updates-testing, etc) they don't get included in composes.

Replying to [comment:17 kalev]:

Replying to [comment:16 ausil]:

how do they get it from cisco? do we ship a .repo file that points them at the right location? what is the interface we have with cisco?

The idea was that we'd ship .repo files pointing users to Cisco's servers. Everything would be signed by our regular keys, so we don't need anything else for verification. If it checks out against our own signing keys, we can trust the build; otherwise package managers would reject it.

One place where we could put the .repo file might be http://pkgs.fedoraproject.org/cgit/fedora-workstation-repositories.git/

I do not know what that is but I think we should put it in fedora-repos, people using xfce, KDE or some other non-workstation desktop fedora experience deserve to have access to the codec.

Right now our interface with Cisco is an email interface, where we'd email them a tarball to upload to their servers. The tarball would contain everything we need; I guess we'd send them 4 repositories/directories for the last releases (F21-F24).

we have no way to exclude the build from composes without writeing code

If we don't tag builds with the special tags (f21, f21-updates, f21-updates-testing, etc) they don't get included in composes.

if they are not tagged they get garbage collected and removed from koji entirely. which we should never do considering that the software will appear to come from us. the best thing to do would be to have some seperate tag that we can manually mash, tar up and make available to them. if we are going to do this I would want us to sign the repo metadata as mirrormanager would not be able to provide verification of the metadata.

Hi, I'm Mike, Koji developer.

I don't see the term 'embargo' in this ticket, but its unclear just how private you want this to be. The htaccess lines above will block access to one method of downloading the rpms, but there will still be a build page that shows the changelog, rpm list, etc.

As Ausil rightly points out, the xmlrpc has a separate download mechanism -- the downloadTaskOutput call. The cli provides a download-task command that will easily download the files generated by a task. This would provide a convenient way around the htaccess blockage. Though, I should note this mechanism depends on the symlinks under the work/tasks directory, which are generally removed after a short period (days or weeks depending on how the admins set things up).

So to summarize:
* folks will easily be able to find out basic information about the build
* folks will be still be able to download the build with a simple cli command

Side note: for koji 2.0 we want to provide finer grained access controls, but I expect you want this capability sooner than that.

If you want to close the xmlrpc hole, you could theoretically do that with hub plugin that wraps the call and selectively denies based on the filename.

Also, how are legitimate folks expected to get the resulting builds? Direct fs access?

Also also, will this build ever be needed as a build dependency? If so, need to consider how the builders will access it to install in the buildroot. Ah, I see kevin pointed this out earlier. Hopefully not ever a dep then.

So, replying to various things:

  • Agree with Dennis that we should have it in fedora-repos for widespread use.

  • Agree that we should use seperate tags for these, mash them and send the entire repos to cisco. Note that we may have to fine tune the redirect for this to allow the masher host to have access to the binary rpms

Replying to mikem:

We only want to not distribute the binary rpms. We are fine distributing the rest of things, including information about the binary rpms, etc. This is NOT a embargo implementation, only a "non binary distribute" hack.

Legit folks get the builds via cisco after we mash and send it to them. But yes, the masher host will need access to them to do that.

Agreed we don't want to restrict repo access to just Workstation users. It seems like this is a good case for a subpackage of fedora-release with repos that people could exclude locally if desired. Also, generally I couldn't see ever having builds of this nature be dependencies. Fedora continues to be self-hosting.

It sounds like we have agreement that tagging with a special tag is sufficient to exclude from composes. So to confirm the implementation details here so we can relay to FESCo, which from what I saw in logs was looking for this kind of solution:

  • Create said tags in koji, e.g. ''f22-exclude'', ''f23-exclude'', and perhaps derivative tags for this specific package as a subset. (Not trying to dictate that, just thinking it would make sense to use inheritance, in case there's ever a need for a similar package in the future.)
  • Set up redirects that make it possible to stop simple web based download of the binary package as kfenzi described above
  • Make appropriate changes to fedora-release to provide repo definition, disabled if desired (my assumption is that this is also a good case for ''enabled_metadata=1'' in the appropriate .repo file)

Is this correct? Is there something else missing?

This all sounds correct to me.

I think next steps:

  • Run plan by FESCo for approval. (should I file a ticket? someone else?)

  • Someone (who?) needs to package up the codec and submit it for review.

  • Someone (who?) needs to review the package.

  • Once added, we need to coordinate with infrastructure and releng and put the redirect and tags in place.

  • Package built.

  • releng needs to mash up the package and sign it and repodata and send the entire thing off to cisco.

I would be happy to take care of putting the package together and submitting it for review once it has gotten FESCo's blessing.

Replying to [comment:23 kevin]:

This all sounds correct to me.

I think next steps:

  • Run plan by FESCo for approval. (should I file a ticket? someone else?)

I'm working on filing the ticket right now. :-)

Replying to [comment:18 ausil]:

if they are not tagged they get garbage collected and removed from koji entirely. which we should never do considering that the software will appear to come from us. the best thing to do would be to have some seperate tag that we can manually mash, tar up and make available to them. if we are going to do this I would want us to sign the repo metadata as mirrormanager would not be able to provide verification of the metadata.

Also, it's ''not'' a foregone conclusion the software "appears to come from us." It will be physically distributed elsewhere. It will be signed by our key. The disabled repo support clearly indicates the software is not distributed by Fedora so the opposite case is easily made.

Closing this ticket as the infrastructure side of this is implemented. I am still working on the fedora-repos and release engineering side of it.

Reopening for infrastructure work. After further discussion with Cisco, here's the plan:

  • Fedora hosts repository directory tree including metadata but excluding RPM packages
  • Cisco hosts RPM packages on their CDN
  • Fedora implements web redirect to point to CDN for actual files
  • fedora-repos points to appropriate Fedora URLs which will be rewritten on request

We need a subdomain in Fedora; one suggestion is codecs.fedoraproject.org, and we can organize with subdirs beneath including something like 'cisco-openh264'.

I've set this up in staging to test things out. :)

https://codecs.stg.fedoraproject.org/openh264/

Please let me know if the naming or setup needs to change any.

Right now all *.rpms redirect to our wiki page about non downloadable rpms. We will need the exact path to redirect them to on the cisco cdn.

Once beta is out and we are out of freeze we can push this to production. :)

I've now pushed this live in production.

The content there is now the current builds and the redirect redirects to the ciscso CDN, so as far as I can tell the infra / repo side is all set now.

We just need:

a repo file added to fedora-repos (best tracked in a bug against fedora-repos?)

a SOP for updating the content (best in the releng docs).

Let me know if you see anything amiss.

Login to comment on this ticket.

Metadata