cloud-sig

Clone
Source Code
GIT

#289 Fedora-Cloud-Base-Rawhide qcow2 - network issue
Closed: fixed 5 years ago Opened 5 years ago by bgoncalv.

Closed: fixed
Reopen Issue

There is some network issue when booting recent Fedora-Cloud-Base-Rawhide qcow2 images.

Latest good image seems to be: Fedora-Cloud-Base-Rawhide-20180615.n.0.x86_64.qcow2

Tested now with: Fedora-Cloud-Base-Rawhide-20180625.n.0.x86_64.qcow2

[^[[0;32m  OK  ^[[0m] Started Login Service.
[^[[0;32m  OK  ^[[0m] Started OpenSSH ed25519 Server Key Generation.
[^[[0;32m  OK  ^[[0m] Started OpenSSH ecdsa Server Key Generation.
[^[[0;32m  OK  ^[[0m] Started OpenSSH rsa Server Key Generation.
[^[[0;32m  OK  ^[[0m] Reached target sshd-keygen.target.
[^[[0;32m  OK  ^[[0m] Started NTP client/server.
[   21.336959] cloud-init[475]: Cloud-init v. 17.1 running 'init-local' at Wed, 27 Jun 2018 13:36:22 +0000. Up 20.48 seconds.
[   21.351550] cloud-init[475]: 2018-06-27 13:36:23,390 - stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']
[^[[0;32m  OK  ^[[0m] Started Initial cloud-init job (pre-networking).
        Starting Initial cloud-init job (metadata service crawler)...
[^[[0;32m  OK  ^[[0m] Reached target Network (Pre).
        Starting Hostname Service...
[^[[0;32m  OK  ^[[0m] Started Hostname Service.
[   23.740381] cloud-init[525]: Cloud-init v. 17.1 running 'init' at Wed, 27 Jun 2018 13:36:24 +0000. Up 22.03 seconds.
[   23.753440] cloud-init[525]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[   23.765779] cloud-init[525]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   23.777763] cloud-init[525]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[   23.790319] cloud-init[525]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   23.802824] cloud-init[525]: ci-info: | eth0:  | False |     .     |     .     |   .   | 52:54:00:12:34:56 |
[   23.816264] cloud-init[^[[0;32m  OK  ^[[0m] [525]: ci-info: |  lo:   |  True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |Started Initial cloud-init job (metadata service crawler).

[^[[0;32m  OK  ^[[0m[   23.840372] ] Reached target Cloud-config availability.
[^[[0;32m  OK  ^[[0m] Reached target Network is Online.

autocloud also detected the issue: https://apps.fedoraproject.org/autocloud/jobs/4634

yeah. i'm able to reproduce the error:

[   11.524044] cloud-init[448]: 2018-06-27 13:47:32,695 - stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']

will ask @gholms and @larsks to investigate

ok thanks to @larsks for pointing out that the version of cloud-init has not changed from 20180615.n.0 to 20180625.n.0. It must be a dependency. Here is the diff between the two (thanks to rpm-ostree for the info): 

rpm-ostree --repo=./ db diff 707bd0e1213f8e713c676a6018e0d7308b217ec757367faaab115a15e02126da 27425faed3d1ae65e883eca95a4c9d05ae6ec6f2530b66ea865c2c117c618f4a
ostree diff commit old: 707bd0e1213f8e713c676a6018e0d7308b217ec757367faaab115a15e02126da
ostree diff commit new: 27425faed3d1ae65e883eca95a4c9d05ae6ec6f2530b66ea865c2c117c618f4a
Upgraded:
  GeoIP-GeoLite-data 2018.04-1.fc29 -> 2018.06-1.fc29
  NetworkManager 1:1.10.6-3.fc29.1 -> 1:1.12.0-0.1.fc29
  NetworkManager-libnm 1:1.10.6-3.fc29.1 -> 1:1.12.0-0.1.fc29
  NetworkManager-team 1:1.10.6-3.fc29.1 -> 1:1.12.0-0.1.fc29
  audit 2.8.3-3.fc29 -> 2.8.4-1.fc29
  audit-libs 2.8.3-3.fc29 -> 2.8.4-1.fc29
  bash 4.4.19-2.fc29 -> 4.4.23-1.fc29
  bash-completion 1:2.7-4.fc29 -> 1:2.8-1.fc29
  buildah 1.2-5.gitfc438bb.fc29 -> 1.2-11.git2ac95ea.fc29
  chrony 3.3-1.fc29 -> 3.3-2.fc29
  containers-common 0.1.31-5.git0144aa8.fc29 -> 0.1.31-6.git0144aa8.fc29
  device-mapper 1.02.147-1.fc29 -> 1.02.148-1.fc29
  device-mapper-event 1.02.147-1.fc29 -> 1.02.148-1.fc29
  device-mapper-event-libs 1.02.147-1.fc29 -> 1.02.148-1.fc29
  device-mapper-libs 1.02.147-1.fc29 -> 1.02.148-1.fc29
  dhcp-client 12:4.3.6-24.fc29 -> 12:4.3.6-25.fc29
  dhcp-common 12:4.3.6-24.fc29 -> 12:4.3.6-25.fc29
  dhcp-libs 12:4.3.6-24.fc29 -> 12:4.3.6-25.fc29
  dracut 047-32.git20180515.fc29 -> 047-34.git20180604.1.fc29
  dracut-config-generic 047-32.git20180515.fc29 -> 047-34.git20180604.1.fc29
  dracut-network 047-32.git20180515.fc29 -> 047-34.git20180604.1.fc29
  elfutils-default-yama-scope 0.172-1.fc29 -> 0.172-2.fc29
  elfutils-libelf 0.172-1.fc29 -> 0.172-2.fc29
  elfutils-libs 0.172-1.fc29 -> 0.172-2.fc29
  fedora-release 29-0.4 -> 29-0.5
  fedora-release-atomichost 29-0.4 -> 29-0.5
  filesystem 3.8-3.fc28 -> 3.9-1.fc29
  gawk 4.2.1-1.fc29 -> 4.2.1-2.fc29
  glibc 2.27.9000-22.fc29 -> 2.27.9000-27.fc29
  glibc-all-langpacks 2.27.9000-22.fc29 -> 2.27.9000-27.fc29
  glibc-common 2.27.9000-22.fc29 -> 2.27.9000-27.fc29
  gnupg 1.4.22-7.fc29 -> 1.4.23-1.fc29
  info 6.5-4.fc29 -> 6.5-6.fc29
  initscripts 9.82-1.fc29 -> 10.00-1.fc29
  iscsi-initiator-utils 6.2.0.874-9.git86e8892.fc28 -> 6.2.0.876-1.git4ef9261.fc29
  iscsi-initiator-utils-iscsiuio 6.2.0.874-9.git86e8892.fc28 -> 6.2.0.876-1.git4ef9261.fc29
  kernel 4.18.0-0.rc0.git9.1.fc29 -> 4.18.0-0.rc1.git2.1.fc29
  kernel-core 4.18.0-0.rc0.git9.1.fc29 -> 4.18.0-0.rc1.git2.1.fc29
  kernel-modules 4.18.0-0.rc0.git9.1.fc29 -> 4.18.0-0.rc1.git2.1.fc29
  libsss_idmap 1.16.1-9.fc29 -> 1.16.2-3.fc29
  libsss_nss_idmap 1.16.1-9.fc29 -> 1.16.2-3.fc29
  libsss_sudo 1.16.1-9.fc29 -> 1.16.2-3.fc29
  lvm2 2.02.178-1.fc29 -> 2.02.179-1.fc29
  lvm2-libs 2.02.178-1.fc29 -> 2.02.179-1.fc29
  oci-systemd-hook 1:0.1.16-1.git05bd9a0.fc29 -> 1:0.1.17-1.git87fb61e.fc29
  openssl 1:1.1.0h-4.fc29 -> 1:1.1.0h-5.fc29
  openssl-libs 1:1.1.0h-4.fc29 -> 1:1.1.0h-5.fc29
  ostree 2018.5-1.fc29 -> 2018.6-3.fc29
  ostree-grub2 2018.5-1.fc29 -> 2018.6-3.fc29
  ostree-libs 2018.5-1.fc29 -> 2018.6-3.fc29
  podman 0.6.3-6.git6bdf023.fc29 -> 0.6.5-2.gite89bbd6.fc29
  policycoreutils 2.8-1.fc29 -> 2.8-3.fc29
  policycoreutils-python-utils 2.8-1.fc29 -> 2.8-3.fc29
  python2-asn1crypto 0.24.0-1.fc29 -> 0.24.0-3.fc29
  python2-audit 2.8.3-3.fc29 -> 2.8.4-1.fc29
  python2-docker 3.3.0-1.fc29 -> 3.4.0-1.fc29
  python2-policycoreutils 2.8-1.fc29 -> 2.8-3.fc29
  python3-asn1crypto 0.24.0-1.fc29 -> 0.24.0-3.fc29
  python3-audit 2.8.3-3.fc29 -> 2.8.4-1.fc29
  python3-docker 3.3.0-1.fc29 -> 3.4.0-1.fc29
  python3-policycoreutils 2.8-1.fc29 -> 2.8-3.fc29
  python3-pytoml 0.1.14-5.git7dea353.fc29 -> 0.1.16-1.fc29
  python3-sssdconfig 1.16.1-9.fc29 -> 1.16.2-3.fc29
  readline 7.0-10.fc29 -> 7.0-11.fc29
  runc 2:1.0.0-32.gitdd67ab1.fc29 -> 2:1.0.0-38.git3ccfa2f.fc29
  skopeo 0.1.31-5.git0144aa8.fc29 -> 0.1.31-6.git0144aa8.fc29
  sssd-client 1.16.1-9.fc29 -> 1.16.2-3.fc29
  systemd 238-8.git0e0aa59.fc29 -> 239-1.fc29
  systemd-container 238-8.git0e0aa59.fc29 -> 239-1.fc29
  systemd-libs 238-8.git0e0aa59.fc29 -> 239-1.fc29
  systemd-pam 238-8.git0e0aa59.fc29 -> 239-1.fc29
  systemd-udev 238-8.git0e0aa59.fc29 -> 239-1.fc29
  tcpdump 14:4.9.2-4.fc29 -> 14:4.9.2-5.fc29
  vim-minimal 2:8.1.042-1.fc29 -> 2:8.1.095-1.fc29

any ideas what could have broke this?

nice diff using ostree, are you able to get the diff between 20180615.n.0 and 20180616.n.0?

20180616.n.0 is the first image that has the network issue.

yep here it is:

$ rpm-ostree --repo=./ db diff 707bd0e1213f8e713c676a6018e0d7308b217ec757367faaab115a15e02126da 1055dea1f99991fb56d5ae9e29cc6ff52fa01970555f82fcc8e929c7f717907f 
ostree diff commit old: 707bd0e1213f8e713c676a6018e0d7308b217ec757367faaab115a15e02126da
ostree diff commit new: 1055dea1f99991fb56d5ae9e29cc6ff52fa01970555f82fcc8e929c7f717907f
Upgraded:
  gnupg 1.4.22-7.fc29 -> 1.4.23-1.fc29
  initscripts 9.82-1.fc29 -> 9.83-1.fc29
  kernel 4.18.0-0.rc0.git9.1.fc29 -> 4.18.0-0.rc0.git10.1.fc29
  kernel-core 4.18.0-0.rc0.git9.1.fc29 -> 4.18.0-0.rc0.git10.1.fc29
  kernel-modules 4.18.0-0.rc0.git9.1.fc29 -> 4.18.0-0.rc0.git10.1.fc29
  podman 0.6.3-6.git6bdf023.fc29 -> 0.6.4-1.gitb43677c.fc29
  runc 2:1.0.0-32.gitdd67ab1.fc29 -> 2:1.0.0-36.gitad0f525.fc29
  vim-minimal 2:8.1.042-1.fc29 -> 2:8.1.055-1.fc29

keep in mind that atomic host dosn't contain the the exact set of packages that are in the Fedora Cloud base image, but a lot of them overlap (including cloud-init + deps) so I figured it would give us some clues
Edited 5 years ago by dustymabe

maybe the initscripts change that went from 9.82->9.83 and has since gone to 10.0? @dkaspar do you think the initscripts changes could have broken the cloud images from booting?

So I've booted up 20180615.n.0 and updated initscripts and I was able to reproduce the error.

dnf install --disablerepo=* https://kojipkgs.fedoraproject.org//packages/initscripts/9.83/1.fc29/x86_64/initscripts-9.83-1.fc29.x86_64.rpm

I've opened BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1596062
Edited 5 years ago by bgoncalv

thanks! please let us know updates from that BZ

I think this issue can be closed Fedora-Cloud-Base-Rawhide-20180719.n.0.x86_64.qcow2 works well for me.
http://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180719.n.0/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-20180719.n.0.x86_64.qcow2

Metadata Update from @dustymabe:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
5 years ago

Login to comment on this ticket.

Metadata
None
None
None
None
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.