#788 rpm from external repo can hit db constraint
Closed: Fixed 6 years ago Opened 6 years ago by tkopecek.

https://koji.fedoraproject.org/koji/taskinfo?taskID=24389979

I suspect, that if we've same rpm internally and in external repo, _setList can hit this constraint. Check the situation and fix constraints/import to handle external repo.


Is it possible that the external repo has remade this rpm?

Possibly a race, if two builds end up adding same external rpm at same time.

The code does attempt to detect duplication before it inserts. I wonder why that isn't working here

The add_external_rpm call is exposed (admin-only) through the api. I can't replicate this by adding an rpm that matches an existing one.

For reference, here is the line from the pkgorigins file from
the repo used in the referenced task:

glibc-all-langpacks-2.26.9000-48.fc28.aarch64   http://kojipkgs.fedoraproject.org/repos/f28-build/latest/aarch64/

I'm betting on a race here. Here is the external rpm entry that conflicted:

https://koji.fedoraproject.org/koji/rpminfo?rpmID=12580194

It was used in a couple other buildroots that were generated very close to this one:

module-80327dd3e2e5d4f5-build-11147326-843063 2018-01-23 08:36:49
module-80327dd3e2e5d4f5-build-11147314-843063 2018-01-23 08:35:34

I'm attaching a script that seems to replicate this race, at least against my local system.
issue788.py

For bonus points, a small tweak of this script generates a deadlock condition

@tkopecek any idea how often this is happening? Note sure how to gauge the priority here

I think it is rare. @jkaluza have you hit it more times than once?

I should note that the above script adds junk data (fake external repo and fake external rpms) to the Koji instance you use for the test, so consider that before using it.

Well the following naive fix does not work:

https://github.com/mikem23/koji-playground/commits/issue788

Can't just catch the insert error and recheck for dups, because the failed insert aborts the transaction.

@tkopecek: We hit it at least three times so far. It is not that rare, but it happens randomly. People were not building lot of modules in Fedora during devconf, so I really cannot say how often it would normally happen. But it is frequent enough to be a problem for us.

Edit: It has been reported three times, it's hard to find out failed koji builds with this error for me. Maybe you have some way how to do that yourself.

@mikem thank you for working on this. Do you have an ETA on an official fix?

Ah, the naive fix works much better if I use a savepoint

@mikem do you have a ticket we can follow to know when this change will be released and/or deployed?

@mprahl: In theory, Patrick has rights to backport this to Fedora Koji. I've created https://pagure.io/fedora-infrastructure/issue/6697 to track this request.

The fix appears to trivially rebase or cherry-pick onto 1.15 (e.g. git cherry-pick -m 1 04635ac)

Metadata Update from @mikem:
- Issue set to the milestone: 1.16

6 years ago

Login to comment on this ticket.

Metadata
Attachments 1
Attached 6 years ago View Comment