PR#370: [WIP] Add SOP that deals with OpenH264 - infra-docs-fpo

infra-docs-fpo

#370 [WIP] Add SOP that deals with OpenH264

Opened a month ago by patrikp. Modified a month ago

patrikp/infra-docs-fpo openh264 into master

Add SOP that deals with OpenH264

Patrik Polakovič • a month ago

c5e2290

modules/releng_misc_guide/pages/sop_generating_openh264_composes.adoc

file removed

-185

		`@@ -1,185 +0,0 @@`
		`- == Generating Openh264 Composes`
		`-`
		`- === Description`
		`-`
		`- Openh264 repos are a special case and we need to generate the composes`
		`- for it in a different way. We use ODCS to generate the private compose`
		`- and send the rpms to Cisco to publish them on their CDN. We publish the`
		`- repodata on our side.`
		`-`
		`- [WARNING]`
		`- .Warning`
		`- ====`
		`- We do not have all the appropriate legal rights to distribute these`
		`- packages, so we need to be extra carefull to make sure they are never`
		`- distributed via our build system or websites`
		`- ====`
		`-`
		`- === Action`
		`-`
		`- ==== Permissions needed`
		`-`
		`- You will need some ODCS permissions in order to request private composes`
		`- and composes from tags. You can set this in infra/ansible in`
		`- inventory/group_vars/odcs in the odcs_allowed_clients_users variable.`
		`- See other releng users entries for format.`
		`-`
		`- ==== Get the odcs token`
		`-`
		`- In order to generate an odcs compose, you need a openidc token.`
		`-`
		- Run the odcs-token.py under `scripts/odcs/` from pagure releng
		`- repository to generate the token.`
		`-`
		`- ....`
		`- $ ./odcs-token.py`
		`- ....`
		`-`
		`- ==== Make sure rpms are written out with the right signature`
		`-`
		`- ....`
		`- $ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38`
		`- ....`
		`-`
		`- Where the key for that branch is listed, then the open264 package and`
		`- version.`
		`-`
		`- ==== Generate a private odcs compose`
		`-`
		`- With the token generated above, generate the odcs private compose`
		`-`
		`- ....`
		`- $ python odcs-private-compose.py <token> <koji_tag> <signingkeyid>`
		`- ....`
		`-`
		- `koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264
		`- tags where [.title-ref]#xx# represents the fedora release)`
		`-`
		- `signingkeyid`: The short hash of the key for this Fedora branch.
		`-`
		- The composes are stored under `/srv/odcs/private/` dir on
		- `odcs-backend-releng01.iad2.fedoraproject.org`
		`-`
		`- ==== Pull the compose to your local machine`
		`-`
		`- We need to extract the rpms and tar them to send them to Cisco. In order`
		`- to that, first of all we need to pull the compose to our local machine.`
		`-`
		`- ===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org`
		`-`
		`- Since the compose is owned by [.title-ref]#odcs-server# pull it into`
		`- your home dir`
		`-`
		`- ....`
		`- $ mkdir ~/32-openh264`
		`- $ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/`
		`- $ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/`
		`- ....`
		`-`
		`- ===== Sync the compose to your local machine`
		`-`
		`- Pull in the compose from your home dir on odcs releng backend to your`
		`- local machine into a temp working dir`
		`-`
		`- ....`
		`- $ mkdir openh264-20200813`
		`- $ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/`
		`- ....`
		`-`
		`- ===== Make the changes needed`
		`-`
		`- Please follow the following commands to make the necessary tar files to`
		`- send to Cisco`
		`-`
		`- ....`
		`- $ cd openh264-20200813`
		`- $ mkdir 32-rpms`
		`- # Copy rpms including devel rpms`
		`- $ cp -rv 32-openh264/compose/Temporary/////*rpm 32-rpms/`
		`- # Copy debuginfo rpms`
		`- $ cp -rv 32-openh264/compose/Temporary//////rpm 32-rpms/`
		`- # copy the src.rpm`
		`- $ cp -rv 32-openh264/compose/Temporary/////*src.rpm 32-rpms/`
		`- $ cd 32-rpms`
		`- # Create the tar file with the rpms`
		`- $ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm`
		`- ....`
		`-`
		`- We need to send this tar file to Cisco along with the list of rpms in`
		`- each tarball.`
		`-`
		`- ===== Syncing the compose to sundries01`
		`-`
		`- Once we get a confirmation from Cisco that the rpms are updated on their`
		`- CDN, verify them by using curl. For example:`
		`-`
		`- ....`
		`- $ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm`
		`- ....`
		`-`
		`- Now push these composes to sundries01.iad2.fedoraproject.org and`
		`- mm-backend01.iad2.fedoraproject.org`
		`-`
		`- On sundries01 we need to sync to a directory that is owned by _apache_,`
		`- so first we sync to the home directory on sundries01. Same with`
		`- mm-backend01 as the directory is owned by _root_.`
		`-`
		`- Create a temp working directory on sundries01`
		`-`
		`- ....`
		`- $ ssh sundries01.iad2.fedoraproject.org`
		`- $ mkdir openh264-20200825`
		`- ....`
		`-`
		`- Create a temp working directory on mm-backend01`
		`-`
		`- ....`
		`- $ ssh mm-backend01.iad2.fedoraproject.org`
		`- $ mkdir openh264-20200825`
		`- ....`
		`-`
		`- Then from your local machine, sync the compose`
		`-`
		`- ....`
		`- $ cd openh264-20200825`
		`- $ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825`
		`- $ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825`
		`- ....`
		`-`
		`- On sundries01`
		`-`
		`- ....`
		`- $ cd openh264-20200825`
		`- $ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/`
		`- ....`
		`-`
		`- On mm-backend01`
		`-`
		`- ....`
		`- $ cd openh264-20200825`
		`- $ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/`
		`- ....`
		`-`
		`- ===== Extra info`
		`-`
		`- Normally that should be it, but in some cases you may want to push`
		`- things out faster than normal, and here's a few things you can do to do`
		`- that:`
		`-`
		`- On mm-backend01.iad2.fedoraproject.org you can run:`
		`-`
		`- ....`
		`- # sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log`
		`- ....`
		`-`
		`- This will have mirrormanager scan the codecs dir and update it if it's`
		`- changed.`
		`-`
		`- On batcave01.iad2.fedoraproject.org you can use ansible to force all the`
		`- proxies to sync the codec content from sundries01:`
		`-`
		`- ....`
		`- # ansible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies`
		`- ....`
		`-`
		`- Mirrorlist servers should update every 15min.`

modules/releng_misc_guide/pages/sop_openh264.adoc

file added

+206

		`@@ -0,0 +1,206 @@`
		`+ == Generating OpenH264 RPMs and sending them to Cisco`
		`+`
		`+ === Description`
		`+`
		`+ OpenH264 RPMs are a special case and we need to handle it with caution.`
		`+`
		`+ The process is as follows:`
		`+`
		+ . We generate and sign the RPMs using `koji dist-repo`.
		`+ . We send the RPMs to Cisco for hosting.`
		`+ . Once we get confirmation from Cisco that they published the RPMs on their CDN`
		`+ we publish the repodata on our side.`
		`+`
		`+ === Legal background`
		`+`
		`+ OpenH264 is a free software library for real-time encoding and decoding video`
		`+ streams in the H.264/MPEG-4 AVC format.`
		`+`
		`+ In 2013 Cisco released both binaries and source code, and pay all royalties for`
		`+ its use to MPEG LA themselves for any software projects that use Cisco's`
		`+ precompiled binaries (thus making Cisco's OpenH264 binaries free to use);`
		`+ any software projects that use Cisco's source code instead of its binaries`
		`+ would be legally responsible for paying all royalties to MPEG LA themselves,`
		`+ however.`
		`+`
		`+ The workaround is that the Cisco binaries are built in Fedora's infrastructure`
		`+ but distributed by Cisco.`
		`+`
		`+ [WARNING]`
		`+ .Warning`
		`+ ====`
		`+ The RPMs must never be distributed via our build system or websites.`
		`+ ====`
		`+`
		`+ === The process`
		`+`
		`+ ==== Permissions needed`
		`+`
		+ . Make sure to have a valid Kerberos token (`kinit`).
		`+ . Relevant FAS group membership.`
		`+ . TODO`
		`+`
		`+ ==== Generating the repository`
		`+`
		`+ Once a request comes in through the RelEng issue tracker to update OpenH264 to`
		`+ a new version we will first need to generate the RPMs and repodata. We do this`
		+ using the `koji dist-repo` utility.
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The arguments that should be passed to the commands will certainly vary from the`
		`+ examples listed below.`
		`+`
		`+ Practice common sense and double check everything.`
		`+ ====`
		`+`
		`+ Run this command:`
		`+`
		`+ ....`
		`+ $ koji dist-repo f43-openh264 31645531 --noinherit --with-src -a aarch -a`
		`+ ppc64le -a s390x -a x86_64`
		`+ ....`
		`+`
		`+ In the case of the above command:`
		`+`
		+ . `f43-openh264` is the tag the builds are tagged into.
		+ . `31645531` is the signing key for the RPMs. The keys for various versions
		`+ can be found https://fedoraproject.org/security[here].`
		+ . `--noinherit` causes only packages tagged into `f43-openh264` to be included,
		+ without inheritance from the parent tag (e.g. `f43`).
		+ . `--with-src` causes the source RPM to be included.
		+ . `-a <ARG>` options are the respective architectures.
		`+`
		`+ After the command finishes the directory should appear`
		`+ https://kojipkgs.fedoraproject.org/repos-dist[here].`
		`+`
		`+ The RPMs are visible through the web frontend, but we have a rewrite rule that`
		`+ redirects any attempt to download them to a wiki page saying why that is not`
		`+ possible.`
		`+`
		`+ In order to get to the RPMs themselves we must do it from a host on which Koji`
		+ is mounted where they will be in the `/mnt/koji/repos-dist/f43-openh264` directory.
		`+`
		`+ For example:`
		`+`
		`+ ....`
		`+ $ ssh compose-branched01.iad2.fedoraproject.org`
		`+ ....`
		`+`
		`+ After you verify that everything needed is there you may clone the directory to`
		`+ your local machine.`
		`+`
		`+ ....`
		`+ $ rsync -avhHP username@compose-branched01.iad2.fedoraproject.org:/mnt/koji/repos-dist/f43-openh264/6555489 /home/username/openh264/2.6.0/f43`
		`+ ....`
		`+`
		`+ ==== Sending the RPMs to Cisco for hosting`
		`+`
		`+ ===== Preparing the tarball`
		`+`
		`+ We now have the necessary directory on our local machine. Now we need to package`
		`+ it into a tarball and send it to Cisco.`
		`+`
		`+ Create a new directory into which we can copy the RPMs and out of which we will`
		`+ create the tarball.`
		`+`
		`+ ....`
		`+ $ mkdir /home/username/openh264/2.6.0/f43/f43_2.6.0_rpms`
		`+ ....`
		`+`
		`+ Copy the RPMs (including the source RPM) from the original directory to the`
		`+ new directory.`
		`+`
		`+ ....`
		`+ $ cd /home/username/openh264/2.6.0/f43/`
		`+ $ cp -rv 6553481////rpm f43_2.6.0_rpms`
		`+ ....`
		`+`
		`+ Create a list of the RPMs that will be sent to Cisco along with the tarball.`
		`+`
		`+ ....`
		`+ $ ls f43_2.6.0_rpms > f43_2.6.0_rpms_list.txt`
		`+ ....`
		`+`
		`+ Create the tarball.`
		`+`
		`+ ....`
		`+ $ tar -cJvf f43_2.6.0_rpms.tar.xz f43_2.6.0_rpms`
		`+ ....`
		`+`
		`+ ===== Sending the tarball to Cisco`
		`+`
		`+ TODO`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ We used to do this through e-mail. However, lately we have started experiencing issues`
		`+ with e-mail filters (can't send tarballs). We are trying to reach Cisco about`
		`+ possible solutions to this issue.`
		`+ ====`
		`+`
		`+ ==== Publishing the repodata`
		`+`
		`+ Once we receive confirmation from Cisco that their CDN was updated we can`
		+ verify it using `curl`.
		`+`
		`+ ....`
		`+ $ curl -I http://ciscobinary.openh264.org/openh264-2.6.0-1.fc43.x86_64.rpm`
		`+ ....`
		`+`
		`+ Now it is time to publish the repodata on our infrastructure.`
		`+ Navigate to the relevant directory on your local machine and sync the`
		+ `dist-repo` directory to your home directory on `sundries01`.
		`+`
		`+ ....`
		`+ $ rsync -avhHP 6556232 sundries01.iad2.fedoraproject.org/home/fedora/username/openh264/2.6.0/43`
		`+ ....`
		`+`
		+ Connect to `sundries01`.
		`+`
		`+ ....`
		`+ $ ssh sundries01.iad2.fedoraproject.org`
		`+ ....`
		`+`
		`+ Navigate to the relevant directory.`
		`+`
		`+ ....`
		`+ $ pwd`
		`+ /home/fedora/username/openh264/2.6.0/43`
		`+ ....`
		`+`
		+ Sync the contents of the directory to `/srv/web/codecs.fedoraproject.org`.
		`+`
		`+ ....`
		`+ $ sudo rsync -avhHP 6556232/. /srv/web/codecs.fedoraproject.org/openh264/43`
		`+ ....`
		`+`
		`+ Make sure the directory is owned by root.`
		`+`
		`+ ....`
		`+ $ sudo chown -R root:root /srv/web/codecs.fedoraproject.org/openh264/43`
		`+ ....`
		`+`
		`+ TODO`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ This section will have to be improved with more details.`
		`+ ====`
		`+`
		`+ In the database:`
		`+ ....`
		`+ update directory set ctime=0 where name like 'ct.org/openh264/43%codecs.fedoraproject.org/openh264/43%';`
		`+ delete from repository where prefix like '%cisco%43%';`
		`+ ....`
		`+`
		`+ In a MirrorManager OpenShift debug container:`
		`+ ....`
		`+ (app-root) sh-5.2$ scan-primary-mirror --category "Fedora Codecs" -d`
		`+ ....`
		`+`
		`+ === Resources`
		`+ . https://src.fedoraproject.org/rpms/openh264[OpenH264 dist-git repository]`