Following on from #1094, here is an attempt at a macro that works differently. Just drop this into /usr/lib/rpm/macros.d/macros.tibbs or whatever and use %constrain_build at the top of a spec. It should automatically fix anything that uses %_smp_build_ncpus internally, which includes the meson macros, %_smp_mflags and through that, %make_build.
/usr/lib/rpm/macros.d/macros.tibbs
%_smp_build_ncpus
%_smp_mflags
%make_build
Edit: pagure seems to still interpret preformatted text. There's a string that's supposed to be "open parenthesis c colon m colon v close parenthesis" but gets turned into some icon. The raw text is at https://www.math.uh.edu/~tibbs/macros.tibbs
# A macro to limit _smp_build_ncpus # # Redefines _smp_build_ncpus depending on various factors # # -c cpus A fixed number of CPUS # -m mem A memory limit per CPU # -v Enables verbose output # # # Calling this macro with "-m N" where N is an integer will determine the # amount of memory in the machine by parsing /proc/meminfo and then set the # number of CPUs to that amount divided by N, rounded down. %constrain_build(c:m:v) %{lua:\ local fedora = require "fedora.common" local verbose = fedora.hasflag("v") local mem_limit = math.tointeger(fedora.readflag("m")) local cpus = math.tointeger(fedora.readflag("c")) \ -- Parse meminfo to find the total amount of memory in the system local function getmem(verbose) local mem = 0 for l in io.lines('/proc/meminfo') do if verbose then rpm.expand("%{warn:" .. l .. "}") end if l:sub(1, 9) == "MemTotal:" then mem = math.tointeger(string.match(l, "MemTotal:%s+(%d+)")) break end end return mem end \ if cpus ~= nil then fedora.explicitset("_smp_build_ncpus", cpus, verbose) elseif mem_limit ~= nil then local mem_total = getmem(verbose) local limit = math.max(1, mem_total // (mem_limit * 1024)) fedora.explicitset("_smp_build_ncpus", limit, verbose) end }
I suppose it would be reasonable to @pmatilai just to see if I'm on the right track for something that lines up with his idea of how this was supposed to work. Note that this is uses the Fedora Lua infrastructure and probably isn't suitable for direct upstreaming but I would happily rewrite it to be more acceptable.
I updated the file at https://www.math.uh.edu/~tibbs/macros.tibbs to:
%constrain_build -c 10
I'll add a bit of error checking and then try to get some wider testing. I welcome any input on the name or ideas for better implementation.
ἐπιθυμία:~❯ rpm -E '%constrain_build -v -c 7 -m 12000' -E '%_smp_build_ncpus' -E '%_smp_mflags' -E '%make_build' -E '%meson_build'|less warning: MemTotal: 65541152 kB warning: Setting %{_smp_build_ncpus} = 5 5 -j5 /usr/bin/make -O -j5 V=1 VERBOSE=1 /usr/bin/meson compile -C x86_64-redhat-linux-gnu -j 5 --verbose
To cut to the chase, yes that is on the right track.Thanks @tibbs for working on this and bothering to check with me.
The original Change was targeted for Fedora 35 (so rpm 4.17) where you don't need fedora specific macros for sane argument handling, so the question is whether you want to have it on all supported Fedora versions right now or just from F35 onwards.
As for the naming, it's not a bad name for what it does. Names are hard.
Ah, I'm not familiar with any improvements to argument handling, so I'll look that up. I'm sure we will want something going back to EPEL7 even if it's just a dummy macro that allows spec compatibility, so I will have to cook up something compatible in either case.
The Lua stuff got quite a face-lift in 4.17, see https://rpm-software-management.github.io/rpm/manual/lua.html#lua-macros
Yeah, I've been playing with it a bit. I should have an updated version soon.
I note the weird thing where you couldn't have a blank line in a Lua macro seems to be gone. No more weird backslashes.
As an aside, I am curious as to whether RPM is moving towards exposing more of the package structure to Lua. One thing I've wanted for a very long time is the ability to get at the main package %description (without the frightening hacks I've used in the past[1]), but it would be very useful if there were just a table containing the main package and subpackage definitions where you could directly access and modify all of the package tags, scriptlet text and the %files list.
%description
%files
1) You can open /proc/self/fd/3 and parse the specfile in to get the description and anything else you might want, but that was too ugly to live.
OK, here is a new version which uses the rpm 4.17 features and none of the Fedora Lua infrastructure: https://www.math.uh.edu/~tibbs/macros.tibbs-newrpm
I also added a warning if no useful -c or -m options are passed, in case someone calls the macro without any arguments.
When I next find some time I'll send a PR for redhat-rpm-config to get this into rawhide for some testing, and if that goes well and there are no objections then I'll send PRs to get this into the release branches and EPEL, and send a PR with documentation as well.
Yes, that's fixed in 4.17. The intent to allow empty lines inside %{} blocks was always there but so was an equally old (some 20+ years) logic error preventing it...
As an aside, I am curious as to whether RPM is moving towards exposing more of the package structure to Lua.
How should I put this... On a high-level intent level, yes the desire to do that is there. The difficulty is in finding a way of exposing that information in a manner that is sustainable to both packagers and rpm developers. Parsing /proc/self/fd is indeed frightening, but so is exposing the various in-progress internal structures to which the spec is parsed into.
I think it'd be more productive to just define a sensible default for that case. At least traditionally by far the most common override is to disable parallelism entirely due to broken makefiles, so having %constrain_build with no arguments simply limit available cpus to 1 would be a constructive way to avoid the difficulty in forming the warning :sweat_smile: (and as for that, there are many hysterical things going on in there, like {} technically suppressing macro argument expansion,but only on one level, it doesn't help when that argument gets passed down to a built-in that always expands its arguments etc :dizzy_face: ) On a related note, you can get the current macro name from %0, not that it helps the escapes though.
Thanks for the info. I really should follow upstream RPM development more closely. I try when I have the time, but I so rarely have the time.
I think it'd be more productive to just define a sensible default for that case.
That's fair. I reworked things to just limit to 1 if no options are passed. I also removed the warning code and the verbosity option since they were only there to give me a bit of useful info while I was remembering how to do this kind of thing. The final macro isn't really complex enough to need it.
The result is at https://www.math.uh.edu/~tibbs/macros.tibbs
I'll go ahead and send a PR for redhat-rpm-config in rawhide now (won't be called macros.tibbs, of course) so that it can get a little testing. If you would like for me to send this upstream, please let me know. Just a hint as to where it should live would be helpful.
As for the weirdness with calling into warn, honestly that seemed the cleanest way to get the output I was looking for. I'm sure there's a horrible mess under the hood but as a general rule I try not to look too deeply into RPM. I did make things nicer with %0 but then realized it was pointless anyway and got rid if it all.
%0
Had trouble filing a PR yesterday due to a pagure problem, but I filed https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/151 today.
Login to comment on this ticket.