Due to the restart of the fedmsg-hub the same builds were enqueued again.

The VirtualBox backend failed to start the vms. It seems a version issue of VirtualBox, we upgraded the VirtualBox to latest, and manually tested that it is working. From tomorrow's compose we should not be having any issue. I am keeping the issue until we see everything work fine.

This is still happening today. I don't think this is an issue with fedmsg-hub, but I could be wrong.

The issue is due to the new boxes that were added. They are also pushing processing the results.

@sayanchowdhury thanks for confirming.

The issue is due to the new boxes that were added. They are also pushing processing the results.

i.e. there were more fedmsg messages getting pushed to production than should have and this was causing issues with autocloud because it was getting extra information about builds that may or may not have existed, etc..

Thanks for the info sayan

can someone (@kushal, @sayanchowdhury , ?) be responsible for tracking down the root cause of this and getting it fixed? This is becoming a pain and is giving us bad information for releasing the next 2W release.

can someone ( @kushal, @sayanchowdhury , ?) be responsible for tracking down the root cause of this and getting it fixed? This is becoming a pain and is giving us bad information for releasing the next 2W release.

Yesterday I fixed both the new boxes and got fedmsg-hub enabled on them. We are going to remove the old production hardware and move these new boxes into production. So this test was done knowingly for yesterday (still today in your timezone). We will take the box out of fedmsg till we put them back as the production servers.

2.5 weeks since initial report and we are still reporting results for 12 tests: https://apps.fedoraproject.org/autocloud/jobs/633

can we get this closed off?

The old boxes has been shutdown and now autocloud is showing the correct number of images.

Going ahead and closing the issue.