#12470 an aarch64 builder is experiencing issues with networking
Closed: Insufficient data 21 days ago by kevin. Opened 22 days ago by abbra.

Trying to build bind-dyndb-ldap in a sidetag. Scratch build succeeds but rawhide build fails with aarch64 builder claiming it cannot reach to any repo to pick up the build dependencies:

https://kojipkgs.fedoraproject.org//work/tasks/5220/130965220/root.log

DEBUG file_util.py:18:  ensuring that dir exists: /var/lib/mock/f43-build-side-108780-58415348-6565257/root/installation-homedir
DEBUG package_manager.py:295:  ['/usr/bin/dnf5', 'builddep', '--installroot', '/var/lib/mock/f43-build-side-108780-58415348-6565257/root/', '/var/lib/mock/f43-build-side-108780-58415348-6565257/root//builddir/build/SRPMS/bind-dyndb-ldap-11.11-3.fc43.src.rpm', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing']
DEBUG util.py:634:  child environment: None
DEBUG util.py:551:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.ehfzxw0q:/etc/resolv.conf']
DEBUG util.py:556:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', 'a254c4cc63af4deda1db810bd10afbcd', '-D', '/var/lib/mock/f43-build-side-108780-58415348-6565257-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.ehfzxw0q:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f43-build-side-108780-58415348-6565257/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', 'builddep', '--installroot', '/var/lib/mock/f43-build-side-108780-58415348-6565257/root/', '/var/lib/mock/f43-build-side-108780-58415348-6565257/root//builddir/build/SRPMS/bind-dyndb-ldap-11.11-3.fc43.src.rpm', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f43-build-side-108780-58415348-6565257/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:459:  Updating and loading repositories:
DEBUG util.py:459:   build                                  ???% |   0.0   B/s |   0.0   B |  00m00s
DEBUG util.py:459:  >>> Curl error (6): Could not resolve hostname for https://kojipkgs.fedoraprojec
DEBUG util.py:459:  >>> Curl error (6): Could not resolve hostname for https://kojipkgs01.fedoraproj
DEBUG util.py:459:  >>> Curl error (6): Could not resolve hostname for https://kojipkgs02.fedoraproj
DEBUG util.py:459:  >>> Curl error (6): Could not resolve hostname for https://kojipkgs.fedoraprojec
DEBUG util.py:459:  >>> Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.x
DEBUG util.py:459:  Repositories loaded.
DEBUG util.py:459:  Failed to resolve the transaction:
DEBUG util.py:459:  No match for argument: autoconf
DEBUG util.py:459:  No match for argument: autoconf-archive
DEBUG util.py:459:  No match for argument: automake
DEBUG util.py:459:  No match for argument: bind-devel >= 32:9.18.35-2
DEBUG util.py:459:  No match for argument: bind-lite-devel >= 32:9.18.35-2
DEBUG util.py:459:  No match for argument: krb5-devel
DEBUG util.py:459:  No match for argument: libtool
DEBUG util.py:459:  No match for argument: libuuid-devel
DEBUG util.py:459:  No match for argument: openldap-devel
DEBUG util.py:459:  You can try to add to command line:
DEBUG util.py:459:    --skip-unavailable to skip unavailable packages
DEBUG util.py:608:  Child return code was: 1

130965220 buildArch (bind-dyndb-ldap-11.11-3.fc43.src.rpm, aarch64): open (buildvm-a64-09.iad2.fedoraproject.org) -> FAILED: BuildError: error building package (arch aarch64), mock exited with status 30; see root.log for more information

I succeeded by re-trying the build which picked up a different aarch64 builder. This ticket is to make sure next builds would not fail on that aarch64 builder.

I assume the best way forward will be to redeploy the builder vm.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

22 days ago

When reporting problems can you provide the top level task? It makes it easier to see what host it is and get to all the log files.

For the record: https://koji.fedoraproject.org/koji/taskinfo?taskID=130965220

The errors seem to be dns related. I don't see any issues with the builder off hand, no errors in dmesg/logs, it seems fine.

So I am not sure what happened here, but I don't think reinstalling will help...

I didn't realized this is issue in the network itself. In that case reinstall will not help.

Maybe it was just some network instability at the time of the build.

Yeah, but it's very strange. We have internal dns servers and they don't seem to have been overloaded. ;(

I guess lets close this and see if it happens any further or was a one off?

@abbra please re-open or file a new one if you ever see this again and we can try and decern some more pattern out of it.

Metadata Update from @kevin:
- Issue close_status updated to: Insufficient data
- Issue status updated to: Closed (was: Open)

21 days ago

Log in to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog