#9091 Very slow s390x builder
Closed: Fixed 3 years ago by kevin. Opened 3 years ago by orion.

Describe what you would like us to do:


This build is taking forever on s390x:
https://koji.fedoraproject.org/koji/taskinfo?taskID=46204875

It's running tests, and for example this test on s390x:

7: [ RUN      ] Size_CvtMode_cvtColor8u.cvtColor8u/320, where GetParam() = (1920x1080, CX_Luv2LBGRA)
7: [ PERFSTAT ]    (samples=100   mean=170.26   median=124.30   min=63.22   stddev=104.29 (61.3%))
7: [       OK ] Size_CvtMode_cvtColor8u.cvtColor8u/320 (100159 ms)

apparently took 100 seconds, but on armv7:

7: [ RUN      ] Size_CvtMode_cvtColor8u.cvtColor8u/320, where GetParam() = (1920x1080, CX_Luv2LBGRA)
7: [ PERFSTAT ]    (samples=10   mean=36.97   median=36.98   min=36.79   stddev=0.07 (0.2%))
7: [       OK ] Size_CvtMode_cvtColor8u.cvtColor8u/320 (1894 ms)

it took 1.9 seconds.

It does seem like tests were running faster earlier in the build.

When do you need this to be done by? (YYYY/MM/DD)


ASAP


I am not sure there is much we can do.. these are shared systems with other jobs being run on the s390 at the same time builds are done. This will affect CPU and other things just like any other 'cloud' machine. From previous reviews.. 10 minutes later its fast and nothing we have done has changed anything.. then some time later its slow again.

in addtion to what smooge said... I looked and there was a eln mass rebuild going on, so on that same builder at the same time was another package building, as well as all the other builders were loaded (they share storage).
We could perhaps lower the weight so they take less jobs, but usually it's not a problem. ;(

I just updated and rebooted all of them... hopefully that will make them happier.

I guess the next time this happens we can and bring in the mainframe people again to see if there is any further tuning they could do.

I'm not sure there's much more to do right now tho... feel free to re-open if you see this again

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata