There seems to be a consistent failure in Azure pipelines run with PR-CI. Multiple re-runs didn't resolve this situation.
The error is
subprocess.CalledProcessError: Command '['hostnamectl', 'set-hostname', 'master1.ipa.test']' returned non-zero exit status 1
as seen here. Currently affected PRs: #6368, #6370, #6342
Metadata Update from @slev: - Issue assigned to slev
systemd-hostnamed with debug loglevel of systemd shows nothing interesting besides tons of remount messages (it seems remounts are very slow or something keeps the resources busy). Note: the remounts are expected and are part of the job of service isolation. So, systemd-hostnamed does it's job but the start up of this service is very slow and client (hostnamectl) failed with timeout (25sec).
systemd-hostnamed
debug
hostnamectl
Probably, performance degradation was caused by new azure agent which was updated last Monday: azure agent 2.204.0 (reply time: 19sec):
1 [2022-07-18 07:29:56] INFO: 1_master_1: run: ['hostnamectl', 'set-hostname', 'master1.ipa.test'] 1 [2022-07-18 07:30:15] INFO: 1_master_1: result: 0
azure agent 2.206.1 (reply timeout: 25sec):
1 [2022-07-21 20:20:04] INFO: 1_master_1: run: ['hostnamectl', 'set-hostname', 'master1.ipa.test'] 1 [2022-07-21 20:20:30] INFO: 1_master_1: result: 1
It's hard to say what was changed, at least nothing suspicious can be found in the changelog.
This issue can be fixed in two ways: - raise timeout for hostnamectl - rearrange jobs
Additionally, there is running clamav on azure agent that is very greedy:
top - 11:15:45 up 1 min, 0 users, load average: 2.28, 0.82, 0.30 Tasks: 159 total, 2 running, 157 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 13.3 ni, 86.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 6943.7 total, 1596.5 free, 1522.8 used, 3824.4 buff/cache MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 5108.9 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3621 root 39 19 865532 796648 7972 R 26.7 11.2 0:08.54 clamscan 1672 vsts 20 0 3538232 124428 57296 S 0.0 1.7 0:04.69 Agent.W+ 1617 vsts 20 0 3439088 113032 52320 S 0.0 1.6 0:01.47 Agent.L+ 3805 root 20 0 1446036 97004 47676 S 0.0 1.4 0:10.92 dockerd 748 root 20 0 3716468 87548 47084 S 0.0 1.2 0:01.66 provisi+
PR: https://github.com/freeipa/freeipa/pull/6376
master:
ipa-4-10:
Metadata Update from @rcritten: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
ipa-4-9:
https://pagure.io/freeipa/c/e77b0b08d78d4d5ae7632ef23aebc577848ea507 https://pagure.io/freeipa/c/98c6e96e8db3d5bdc0315094b8a7bf81d196479b https://pagure.io/freeipa/c/b59baf31bc097821ff7787ecd75affb27ea2a7c3 https://pagure.io/freeipa/c/1ada42e3bce58a729e689377b1a41b6cfa90b508
Login to comment on this ticket.