Meh. Seems like some serious problem in our handling of shared variables
for multiprocessing ..., dunno how 1145038 could be changed to 1145064.
/usr/bin/copr-rpmbuild --verbose --drop-resultdir --srpm --build-id 1145064 --detached

https://copr.fedorainfracloud.org/coprs/g/python/python3.9/build/1145064/ failed with:

Wrote: /var/lib/copr-rpmbuild/resultszjwar3zt/astrometry.spec
setting SOURCE_DATE_EPOCH=1570060800
Wrote: /var/lib/copr-rpmbuild/resultszjwar3zt/astrometry-0.78-5.fc31.src.rpm
stderr: 


Traceback (most recent call last):
  File "/usr/bin/copr-rpmbuild", line 129, in main
    action(args, config)
  File "/usr/bin/copr-rpmbuild", line 210, in build_srpm
    produce_srpm(task, config, resultdir)
  File "/usr/bin/copr-rpmbuild", line 159, in produce_srpm
    shutil.copy(os.path.join(tempdir, item), resultdir)
  File "/usr/lib64/python3.7/shutil.py", line 248, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.7/shutil.py", line 122, in copyfile
    copyfileobj(fsrc, fdst)
  File "/usr/lib64/python3.7/shutil.py", line 82, in copyfileobj
    fdst.write(buf)
OSError: [Errno 28] No space left on device

So it looks like 1145064 ended up in weird state, we recycled the builder for
build 1145038 - and because of some bug related to pr #721 we did not
even start the next build and downloaded previous data.

We are not able to fix this till tomorrows release; sorry. It doesn't seem to
be something which should bite you again - elsewhere then in large
astrometry.

IOW, please update this ticket if you happen to see the problem again.
Thank you.

I believe that the confused build should always be failed, because the trigger
of this issue seems to be the failure of previous build. (so you shouldn't
see succeeded status with wrong results).

OK. I'll try to stop building astrometry. cc @cstratak

I guess this probably only happens on AWS builders, where we have pretty small / partition,
and /tmp + /var/lib/copr-rpmbuild/ beneath.

I can fix those images, but I'll have to re-generate our builder VM images. I hope I'll
get to that tomorrow.

For the real fixes -- we need to implement "system probes" in copr-rpmbuild which
will warn us when we regressed back to this bug. Those should tell "hey, you have
small /tmp, or small /var/lib/copr-rpmbuild".

Plus, we for some reason don't clean everything from /tmp ... there are many
leftovers on @churchyard's builders because those are heavily recycled. This
needs precise exception handling ... I hope I'll get to this soon in #721.

Also, we should rather sooner implement some nicer VM killing policy, e.g.
configure some "max_builds_count" per VM group. Also we could terminate
the builder after say 5 consecutive build failures.

Another case here:

https://copr.fedorainfracloud.org/coprs/g/python/python3.9/build/1220089/

This says clone url is https://src.fedoraproject.org/rpms/python-markdown_2.git but the log suggest the package is kicad and there is a No space left on device problem again.

Those 2 also say kicad:

https://copr.fedorainfracloud.org/coprs/g/python/python3.9/build/1220075/
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/build/1220063/

yeah, fortunately it failed again without further poisoning the chroot ...
This should be fixed in:
https://pagure.io/copr/copr/pull-request/1245

Commit ad5422e fixes this issue