#7585 Trigger another F28 Atomic Host nightly compose for 20180625 due to Inconsistency in OSTree version across arches
Closed 3 years ago Opened 3 years ago by sinnykumari.

Existing twoweek nightly compose from 20180625 contains OSTree version 28.20180623.0 for both aarch64 and ppc64le media while x86_64 has OSTree version 28.20180625.0 .
Not sure what could have caused this problem because I didin't find anything useful in compose log.
I ran imagefactory locally on ppc64le around 1 hour back with config from ppc64le createImage task and OSTree version was fine (28.20180625.0) .

Please trigger another F28 AH twoweek nightly compose 20180625.1 so that we can use it for twoweek RC for next release if everything looks good.


This looks like a race condition where the updates compose was running at the same time as the twoweek nightly atomic run.

updates compose ostree tasks started at 08:48:45 and ended at 09:27:07:

2018-06-25 08:48:45 [INFO    ] [BEGIN] ---------- PHASE: OSTREE ----------
...
2018-06-25 09:27:07 [INFO    ] [DONE ] ---------- PHASE: OSTREE ----------

twoweek compose image build tasks started at 09:17:42 and finished at 09:43:04:

2018-06-25 09:17:42 [INFO    ] [BEGIN] ---------- PHASE: IMAGE_BUILD ----------
...
2018-06-25 09:43:04 [IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: qcow2-raw-xz, arches: aarch64-ppc64le-x86_64, variant: AtomicHost, subvariant: AtomicHost) (task id: 27853671)

mark this as another reason we want to generate the ISO and VM images inside the updates compose and not as a separate compose.

This looks like a race condition where the updates compose was running at the same time as the twoweek nightly atomic run.
updates compose ostree tasks started at 08:48:45 and ended at 09:27:07:
2018-06-25 08:48:45 [INFO ] [BEGIN] ---------- PHASE: OSTREE ----------
...
2018-06-25 09:27:07 [INFO ] [DONE ] ---------- PHASE: OSTREE ----------

twoweek compose image build tasks started at 09:17:42 and finished at 09:43:04:
2018-06-25 09:17:42 [INFO ] [BEGIN] ---------- PHASE: IMAGE_BUILD ----------
...
2018-06-25 09:43:04 [IMAGE_BUILD ] [INFO ] [DONE ] Creating image (formats: qcow2-raw-xz, arches: aarch64-ppc64le-x86_64, variant: AtomicHost, subvariant: AtomicHost) (task id: 27853671)

Ah I see.
I wonder if it was a race condition, why did it pick OSTree version 28.20180623.0 instead of 28.20180624.0 ?

mark this as another reason we want to generate the ISO and VM images inside the updates compose and not as a separate compose.

I wonder if it was a race condition, why did it pick OSTree version 28.20180623.0 instead of 28.20180624.0 ?

I have no idea. It's really weird the 0624 compose is marked as finished but there is no 0624 commit in the repo (for x86_64):

commit fbed0e26736fc189129f80e9547bfd71497377ca49f0cfd421f173667f5ea825                                                                                                                                           
ContentChecksum:  c558c8d4f54308b71302a587ef8f65fa659dfca0e31f94d8a52f2d3a5fe9a08f                                                                                                                                
Date:  2018-06-25 08:58:55 +0000                                                                                                                                                                                  
Version: 28.20180625.0                                                                                                                                                                                            
(no subject)                                                                                                                                                                                                      

commit 5cd042553d8c2f5d49f91630279eaf29af30ffca99ce42bf9ee14bc4c233fdcd                                                                                                                                           
ContentChecksum:  1dc12f3d042ffbb4d1a3f485ebe2de9ff3fc43501fc14797c689f7009fcef75c                                                                                                                                
Date:  2018-06-23 18:55:35 +0000                                                                                                                                                                                  
Version: 28.20180623.0                                                                                                                                                                                            
(no subject)

and from the 0624 compose log:

2018-06-24 19:38:21 [INFO    ] [DONE ] OSTree phase for variant Everything, arch x86_64 (task id: 27842011)
2018-06-24 19:42:17 [ERROR   ] Unable to find commitid file

looks like it couldn't write the commit to the commitid file (which is what robosig uses to update the ref after it signs the commit).

We probably need to investigate this more.

The commitid file is written by rpm-ostree via the --write-commitid-to option. Hoever looking at logs from rpm-ostree there were no changes, hence no commit and thus no commit id was written. This is probably a bug in Pungi. What do you expect should happen in such case? (I expect it should do nothing.)

The commitid file is written by rpm-ostree via the --write-commitid-to option. Hoever looking at logs from rpm-ostree there were no changes, hence no commit and thus no commit id was written. This is probably a bug in Pungi. What do you expect should happen in such case? (I expect it should do nothing.)

ahh I see. We need to add the force_new_commit option to the cofig. @sinnykumari - can you do that?

The commitid file is written by rpm-ostree via the --write-commitid-to option. Hoever looking at logs from rpm-ostree there were no changes, hence no commit and thus no commit id was written. This is probably a bug in Pungi. What do you expect should happen in such case? (I expect it should do nothing.)

ahh I see. We need to add the force_new_commit option to the cofig. @sinnykumari - can you do that?

Yup, I will work on it.

OSTree version from Fedora-Atomic-28-20180625.1 compose looks good on all arches. They all have version 28.20180625.0

Metadata Update from @sinnykumari:
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata