Yesterday I've been fixing staging Koji so that incomplete builds
that in the meantime (after sync) were fixed in production could
be re-imported in staging.

Below I described some problems I found and proposed solutions.
I think that sync script should be improved to avoid such problems
in future - I can provide patch if my solutions seem acceptable.

Problem nr 1: incomplete builds are on prod volume

Incomplete (failed, cancelled or deleted) builds are still on prod
volume. When trying to rebuild or import them, koji will try to write
them on prod volume (and obviously fail because of read-only
filesystem).

Proposed solution: don't set volume to prod for incomplete builds,
leave them on default (staging) volume, with no files in filesystem.

Problem nr 2: incomplete builds have rpms in DB

Incomplete builds can have RPMs in DB. This can happen when
production pg_dump is taken when a build was about to complete - it
already had some RPMs attached, but was still building. Sync script
marks such builds as failed, but it doesn't remove files from DB.
Trying to import such builds later fails with constraint violations
(duplicate RPM NVRA in DB).

Proposed solution: delete RPMs of incomplete builds from DB - they
shouldn't be referenced anywhere (eg. not used in buildroots), so this
should be safe.

Problem nr 3: stale files in /mnt/fedora_koji

This is quite simple IMHO. All old files in /mnt/fedora_koji must
go away during sync. Otherwise problems will be inevitable.