releng

#6645 Make sure OSTree repos are sync'ed correctly after the masher completes
Closed: Fixed Opened by jlebon.

Closed: Fixed
Reopen Issue

We have internal testers which essentially check for new commits by polling the fedora-atomic/25/x86_64/updates/docker-host ref on https://kojipkgs.fedoraproject.org/atomic/25. When the ref changes, a new test is triggered. We've been seeing failures in which the ref gets updated, but the commit object to which the ref points to is not yet in the repo.

We need to make sure that all the refs are being updated safely: first objects, then refs. There is a more complete rsync-repos script that knows how to safely do this.

For the updates ref in specific, judging from the masher code, it seems like we're operating on the live repo and then "waiting" for it sync out?

https://github.com/fedora-infra/bodhi/blob/1a2871520d719c44f5cab37b2a5554741ec30ae7/bodhi/server/consumers/masher.py#L335

https://github.com/fedora-infra/bodhi/blob/1a2871520d719c44f5cab37b2a5554741ec30ae7/bodhi/server/consumers/masher.py#L744

What kind of lag are we talking about? 10m? 30m? Is there any way to know when all the files have been synced out?

We might need to change this so that we compose on a copy of the repo, and then e.g. use the rsync script above to update the real repo. But this presumes that the order in which we rsync to the mount reflects the order in which it hits the public, which may not be true it seems?

Maybe an easy way to do this is to use --write-commitid-to= so that the compose doesn't actually update the ref, and then only update the ref once we hit mashtask.sync.done?

OK, I was able to do some measurements. From the time the ref is updated to the time the object is accessible is around 30 minutes. That's a massive window which will realistically affect not just machines that continuously poll, but users following the updates stream as well.

@jlebon can you schedule some time for me + you + @puiterwijk to talk on this subject.

Metadata Update from @ausil:
- Issue tagged with: meeting

https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bodhi2/backend/files/fedora-updates-push#n52
is the script and where it syncs. This script runs every 30min (at :15 and :45).

But I don't understand why there would be a 30min delay. The script should have everything synced after it runs and we copy objects first and then refs.

Oh, of course bodhi could be syncing into place while the above cron is trying to sync out to master mirrors, resulting in partial doom.

We can/should replace the cron with a fedmsg driven trigger for syncing that runs only when bodhi is done with something and ready for it to sync out.

Right, we had a meeting about this with @puiterwijk a week or two ago, and I think in a nutshell what we discovered was that it was most likely because the cron takes time to sync the objects, and by the time it gets to rsync'ing the ref, another compose might have run which updated it.

We can/should replace the cron with a fedmsg driven trigger for syncing that runs only when bodhi is done with something and ready for it to sync out.

Yeah, that makes sense.

Here is how I implement ostree rsync's to properly fence the refs.

rsync -rlpt --include=/objects    --include=/objects/** --include=/deltas  --include=/deltas/**    --exclude=*  SOURCE/ostree/repo/ DESTINATION/ostree/repo/ --ignore-existing
rsync -rlptvPH --include=/refs    --include=/refs/**    --include=/summary --include=/summary.sig  --exclude=*  SOURCE/ostree/repo/ DESTINATION/ostree/repo/ --delete
rsync -rlptvPH --include=/objects --include=/objects/** --include=/deltas  --include=/deltas/**    --exclude=*  SOURCE/ostree/repo/ DESTINATION/ostree/repo/ --ignore-existing --delete

https://github.com/ostreedev/ostree-releng-scripts/blob/master/rsync-repos is an implementation of this - I'd really like if we made use of ostree-releng-scripts rather than duplicating it.

Someone from infra or releng has to replace the cron job with a fedmsg driven job.

Metadata Update from @mohanboddu:
- Issue untagged with: meeting

Metadata Update from @ausil:
- Issue assigned to mohanboddu

@mohanboddu will check on this and report back. Need to see if cron job is still required.

@mohanboddu reports the work is done. Closing this ticket.

Metadata Update from @syeghiay:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.