Issue #9391: rsync the directory tree to mm-backend01 - releng

releng

#9391 rsync the directory tree to mm-backend01

Closed: Fixed 4 years ago by mohanboddu. Opened 4 years ago by bcotton.

Describe the issue
Enabling the Cisco repo by default in Fedora 32 Workstation results in a dnf error. We think the best approach is to use MirrorManager to handle the repo metadata. This request enables that as described in BZ 1768206c24
When do you need this? (2020/04/16)
If we cannot complete your request, what is the impact?
A release blocking bug is not resolved.

opening on behalf of @adrian

I just saw that it would also be possible to trigger the sync from mm-backend01 as mm-backend01 already has access to the proxies via the mirrormanager user.

The main question for me right now is from where is the sync triggered. It would also be good to know how often MirrorManager has to scan for changes or if we can also trigger the scan of this repository (umdl) somehow.

kevin commented 4 years ago

So, @mohanboddu and I discussed this today and thought:

Short term (since we are going into freeze and since the h264 repo is currently manually created, but will be made by ODCS hopefully later) we just manually sync the repodata to mm-backend01 whenever we update the repo. Where would be good for this? /srv/web/whatever ? We could at the same time trigger mm to re-read it? So, it would need an initial setup to see it, but then after that we only tell it to update when we update the repodata.
Longer term after the release and the datacenter move and ODCS being used, we setup something more automated.

Sound ok? What dir makes sense to put the repodata? I guess we can sync the current repodata from the current live repo. ;)

Thoughts?

kevin commented 4 years ago

Sorry, I missed your question:

I just saw that it would also be possible to trigger the sync from mm-backend01 as mm-backend01 already has access to the proxies via the mirrormanager user.

The way it works now is:

releng runs pungi manually on compose-x86-01 to create the repos.
rpms are sent to cisco
once they update we sync the compose to sundries01
rsync cron jobs sync that to each proxy

For now I have also synced the content to /srv/codecs.fedoraproject.org/ on mm-backend01 for you.

adrian commented 4 years ago

It seems to work:

$ curl -s "https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-32&arch=x86_64" 
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Thu, 09 Apr 2020 10:22:49 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
 <files>
  <file name="repomd.xml">
   <mm0:timestamp>1584475845</mm0:timestamp>
   <size>3074</size>
   <verification>
    <hash type="md5">e2f94cd477d08d8c05a435fcdb2d723d</hash>
    <hash type="sha1">26cb3a40f8f828769d8fed09af7d0631c74573e7</hash>
    <hash type="sha256">5c95a431fa860c69867cbdff65d2e3506c9ca6cb4670223d40cdb3e657487c2a</hash>
    <hash type="sha512">de714821454a94c6e6be3ce765ffa2764c8b458c3f625c9db7e904ccfbd7cca93b02995873473d893e8a30a3868e57cac83f1db1ea4179f985ee484640f94321</hash>
   </verification>
   <resources maxconnections="1">
    <url protocol="https" type="https" location="US" preference="100">https://codecs.fedoraproject.org/openh264/32/x86_64/repodata/repomd.xml</url>
   </resources>
  </file>
 </files>
</metalink>
$ curl -s https://codecs.fedoraproject.org/openh264/32/x86_64/repodata/repomd.xml | sha256sum 
5c95a431fa860c69867cbdff65d2e3506c9ca6cb4670223d40cdb3e657487c2a  -

As always it required small code changes for the repository detection, which I will soon bring upstream.

From my point of view this should be usable/testable now.

I removed the baseurl line from my fedora-cisco-openh264.repo file and added:
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-$releasever&arch=$basearch

and a dnf reinstall openh264 worked (F32, x86_64).

We still need to figure out how this can be automated. Right now even UMDL has to be run manually.