#164 aarch64 builders cbs Cannot allocate memory
Closed: Fixed 3 years ago by arrfab. Opened 3 years ago by fmount.

Hi,
as previously reported in #155, I'm still unable to help building Ceph packages due to a "Memory exhausted" error when reaching target aarch64 [1].
I didn't submitted to cbs more than 1 job, and I double checked the issue by running a --scratch build using just the target in $subject (to make sure parallel tasks didn't hurt since it should be the same hypervisor) [2].
In [1] [2] you can find all the info.

[1] https://cbs.centos.org/koji/buildinfo?buildID=31533
[2] https://cbs.centos.org/koji/buildinfo?buildID=31533


Hi,
as previously reported in #155, I'm still unable to help building Ceph packages due to a "Memory exhausted" error when reaching target aarch64 [1].
I didn't submitted to cbs more than 1 job, and I double checked the issue by running a --scratch build using just the target in $subject (to make sure parallel tasks didn't hurt since it should be the same hypervisor) [2].
In [1] [2] you can find all the info.

[1] https://cbs.centos.org/koji/buildinfo?buildID=31533
[2] https://cbs.centos.org/koji/buildinfo?buildID=31533

Sorry wrong paste in the link section:

[1] https://cbs.centos.org/koji/buildinfo?buildID=31533
[2] https://cbs.centos.org/koji/taskinfo?taskID=1713174

Metadata Update from @arrfab:
- Issue assigned to arrfab

3 years ago

Metadata Update from @arrfab:
- Issue tagged with: cbs, centos-common-infra, low-gain, low-trouble

3 years ago

Metadata Update from @fmount:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)

3 years ago

Issue status updated to: Open (was: Closed)

3 years ago

@fmount : it really depends if there are some other tasks running afaics .. but as you can imagine it's "fire, fire, fire" everywhere due to recent announces ... I'll still chase after some aarch64 capacity to move to other host the CBS aarch64 builders.
OTOH, maybe Ceph needs some kind of memory/core ratio, and even if I bumped memory to max I could in #155 , it's possible that because I also bumped number of vcpus, we suffer from same issue. As a quick workaround, I can at least shutdown/reduce vcpus number and start with same amount of memory.
Then you can quickly give it a try (even a --scratch build would be fine). Faster and easier on my side due to limited available time for this

I agree @arrfab, the spec file can be probably optimized to properly work with some kind of memory/core ratio, in any case if you want me to give a quick test (--scratch) after you put in place the workaround, let me know.
I understand your time on this is limited, so I would work w/ you on this w/ lower priority.

Machines were moved and with more memory but it seems same issue .. wondering if that's specific to aarch64 or something else. Worth investigating ?

Hi @arrfab, thanks a lot for your help so far.
Analyzing the problem we're seeing here, I think that's definitely not enough just adding more and more resources to the builder, but we should have some sort of "Conservative" mode that can help packaging Ceph on aarch64 (reducing the -j factor adding more conditions); by doing this at Makefile level we can help Ceph builds working better no matter the resources provided by the builder machine.
Being that said, feel free to close this ticket as you already have increased the available resources to the maximum amount, and I'll keep you posted on the next steps and checks on Ceph side.

thanks for the feedback, closing so for now but feel free to open if we need some infra changes. As said, "amusingly" it started to show up with recent Ceph versions (but not previous versions) and no other aarch64 build from other SIGs are facing the the issue we see here. So yes, worth investigating in the Makefile (eventually patched through .spec with conditional around aarch64 arch check ?) . Closing so for now

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Issue status updated to: Open (was: Closed)

3 years ago

Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed

3 years ago

Login to comment on this ticket.

Metadata
Boards 2
CBS Status: Done