That's because build environments are different and cbs needs proxy to reach external resources like gitlab, kojihub.stream etc .. so it's set directly on the builders for all SIGs
So whatever can build from within RH firewall can fail on a community build env (which itself has no default route to internet for obvious reasons)

During the build phase, there isn't supposed to be network access anyway, so setting these variables at this point shouldn't be required.

But it's still needed to download required pkgs listed as BuildRequires: in the .spec, and these are external repositories (CentOs Stream, epel, etc) ..

These downloads do not happen during the rpmbuild invocation. When rpmbuild runs, /etc/resolv.conf typically does not even list any name servers, no name service runs on 127.0.0.1, and DNS lookups fail.

@fweimer : do you have suggestions ? as we need the http_proxy for the kojid builders to have a way to retrieve sources from gitlab.com and other external repositories and so the whole kojid process is started with env set for it.

@tkopecek : do you have opinions about that ?

@fweimer : do you have suggestions ? as we need the http_proxy for the kojid builders to have a way to retrieve sources from gitlab.com and other external repositories and so the whole kojid process is started with env set for it.

I'd suggest to set the proxy through dnf.conf. That probably only leaves centpkg-minimal, assuming that the lookaside cache has to go through a proxy as well. @bstinson Any ideas for that?

@fweimer : we tried initially to only set it up through dnf.conf but then only dnf/yum obeys it but not git itself, needed to checkout sources from gitlab.com (external) so reason why it's the whole systemd kojid service that is started with these env variables : https://github.com/CentOS/ansible-role-kojid/blob/master/templates/systemd-kojid-proxy.conf.j2 (boolean that will or not configure it)

Still waiting for feedback from @tkopecek about ensuring that there is a way at the rebuildSRPM step that would use a proxy to retrieve source, build the src.rpm and then only the proxy to retrieve BuildRequires: but nothing else

The only workaround I eventually see for this would be to modify centpkg-minimal to (ab)use http_proxy and https_proxy to retrieve sources and so wouldn't need the whole kojid service inheriting these environment variables ?

I take that back : centpkg-minimal just has get_sources.sh in which we can set https_proxy, but it's too late as it's kojid that does the git checkout before so it's really at the kojid side that it needs to be implemented (and reason why we used the systemd override to have it working)

"stupid idea" : as gitlab.com and other external repositories that we need to reach are behind https, would that work if we were to just set https_proxy env variable and not http_proxy ?

Sorry, missed the ping here. There are two distinct points in code:
1) git checkout - this would be affected by http(s)_proxy for kojid
2) srpm build inside mock - it would be affected by mock.config. If you set up yum_proxy in kojid.conf you should be fine. Nevertheless, it will be used for all builds on that builder.

@tkopecek yes, and that's the problem : as kojid needs https_proxy set for the git checkout, it's then inherited for all tasks, and that's what @fweimer is complaining about : some python tests see the http_proxy (and https_proxy) env variables and one build that work on an internal koji builder doesn't on another koji builder (different setup) that itself needs the proxy for the git checkout part .... You confirm that there is nothing inside koji code that can work for this scenario ? if not we can create a RFE probably ?

Ah, interesting. Yes, in koji is nothing what could help. If we're boiling down to jsut git - what about git config --global http.proxy under kojibuilder user? (or even system)

that's an interesting option but it's not only git as retrieving sources uses curl so we'd still need to find a solution for that , and also kojibuilder doesn't have a real homedir either ...

The reason why we never just allowed plain ip routing is that gitlab itself is fronted by Cloudflare, so adding route to all cloudflare would mean builders having access to (almost) all internet, which is something we refused to do (see https://docs.gitlab.com/ee/user/gitlab_com/#ip-range)

For homedir --system git config could be used.
Anyway, you're right with tarball download. As is centpkg-minimal here, easiest would be to hack this package directly. Passing it through koji would be a weird case - it would be just one function which is passed. What about extending centpkg-minimal call to centpkg-minimal sources --proxy https://xyz. Then it could be managed by scm checkout policy?

This is tricky, because you want the proxy for all the outside-chroot scm calls, and you want it for the inside-the-chroot source_cmd calls, but you don't want it for any of the other chroot stuff.

While tweaking the source_cmd to somehow include the setting might be part of the solution, that won't help with the outside-chroot calls.

Hypothetical

1. leave the global proxy env settings as they are on the host (outside-chroot calls continue to work)
2. set proxy options to blank string in mock site-defaults.cfg. E.g. config_opts['http_proxy'] = '' (keep this problematic setting out of build envs)
3. separately apply the proxy setting in source_cmd either by directly embedding them in the command, or using a modified centpkg-minimal that knows to use the proxy (inside-chroot source fetches continue to work)

separately apply the proxy setting in source_cmd either by directly embedding them in the command, or using a modified centpkg-minimal that knows to use the proxy

It doesn't look like it is possible to embed environment variables in the source_cmd setting, so this would need to be handled within whatever script is used there (unless the script maybe could take an option for this setting).

yesterday during infra prioritisation call we discussed this one as there are other tasks pending for CentOS Stream infra (internal) so I'd need to know if I have/can work on this or not (is that blocking and so waiting for guidance about if I can just work on it or not)
I think we can write something that would let us :

• configure git settings to force going through proxy
• same for koji and declare proxy only for rpm transactions
• modify get_sources.sh to verify env and automatically set http_proxy just for the curl command to retrieve sources from lookaside cache
• unset http_proxy and https_proxy completely everywhere else

What's the priority on this ticket ?

The ISA SIG has a robust workaround in redhat-rpm-config. We need to patch redhat-rpm-config anyway, so it is not particular burdensome. I suspect that it's kind of hard to map the testsuite failures to this particular configuration, so other rebuilding SIGs might be impacted in the future.

Regarding options, maybe you could consider using transparent proxies, iptables DNAT and a reverse or transparent proxy (that doesn't need HTTP proxy headers in the incoming request).

transparent proxies would mean impersonating gitlab.com and other ones at the TLS level, so clearly not an option :/

transparent proxies would mean impersonating gitlab.com and other ones at the TLS level, so clearly not an option :/

No, this isn't a problem. You would only be redirecting the TCP bytestream, which is not authenticated. Basically, you need a program that takes in incoming TCP stream and performs protocol translation for whatever proxy protocol you use (SOCKS, HTTP CONNECT, or just send the request to the right network address through a tunnel). The encrypted HTTP request beneath TLS can remain unchanged and the server identity does not change, just like when HTTPS is tunneled over a HTTP CONNECT request. Request rewriting may be required for the unencrypted HTTP case, to reach the destination through an HTTP proxy which is not configured as a transparent/reverse proxy.

Do you know any transparent tcp proxy that would automatically discovers backend nodes (so not haproxy) and that would accept an ACL to verify domain/fqdn in request to then let it pass through or deny it ? (we don't want to open builders to whole internet and gitlab itself is behind cloudflare, so we can't just bindly open all to cloudflare network)

@bstinson : I can try having a look but it seems you already modified centpkg-minimal (https://cbs.centos.org/koji/buildinfo?buildID=44290) but nothing landed in git (https://git.centos.org/rpms/centpkg-minimal/commits/c8-sig-infra) .. how do you want to proceed ? Ideally I'd like to submit something to git once tested but your locally built package should be backported to git for that : can you have a look ?

Thanks to a direct chat with @fweimer , I was pointed to the workaround they used for the ISA sig builds : injecting this through redhat-rpm-config macros :

# ---- Build policy macros.

# CBS Koji sets these environment variables, which cause compatibility problems.
# <https://pagure.io/centos-infra/issue/1189>
%__spec_build_pre %{___build_pre} \
  unset http_proxy; unset https_proxy
%__spec_check_pre %{___build_pre} \
  unset http_proxy; unset https_proxy; unset no_proxy

Documented here but it's also visible on their pkg on gitlab

We agreed to investigate also with other people involved into different koji instances to find a common solution that would work for all environments, but ISA SIG isn't blocked anymore, thanks for that macro workaround (documented here for other SIGs in case they'd hit the same issue)

While revisiting other ticket for other env, we found a solution that will eventually make it into koji 1.34 (Thanks again @tkopecek !).
Underlying solution was applied to cbs koji env so http_proxy and https_proxy are now gone completely, and shouldn't interfere with builds