Learn more about these different git repos.
Other Git URLs
Seeing a lot of memory allocation failure errors for image builds in f26:
memory allocation failure
Not sure what this is about. I ran an image build locally and it succeeded.
according to @puiterwijk we will need this upstream commit pulled back into our oz and patched on our servers. see the upstream PR for comments.
Metadata Update from @mohanboddu: - Issue assigned to imcleod
updated the title to be more descriptive
buildhw-03.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-08.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-04.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-02.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-07.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-06.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-05.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-10.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-01.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-09.phx2.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- buildhw-aarch64-01.arm.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- 120 factory-build-e453c145-6a28-4591-abd5-d1bf508d6d15 running 121 factory-build-0f9b5bae-b13c-4ec4-98c9-b75cfc8bf513 running 124 factory-build-93c8030d-f8f9-4337-948d-14dc05793963 running 128 factory-build-c5d1f408-fef2-402a-bb25-19ae067e5089 running 135 factory-build-06a229f7-bdd3-4c8e-9449-c85b6fe04112 running 136 factory-build-2ad50b68-db21-4055-8faf-9dff6515354f running buildhw-aarch64-03.arm.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- 115 factory-build-af4645ee-29f1-4a13-8031-587ca6453d9e running 117 factory-build-cafbca2d-db0c-4ffe-b523-745eba60ee2f running 118 factory-build-208c68b2-53f3-4056-b94d-229682822cd2 running 119 factory-build-0dab4805-e819-4d6f-b513-f8b3629241bb running 120 factory-build-7aa92663-372f-427f-ad30-45724bcca6a3 running 121 factory-build-b91c7f85-4612-4219-a9e0-1f99c87fa977 running 123 factory-build-99869671-9156-41d2-82c6-2663ff44858e running 126 factory-build-49f691c6-685a-4abd-8ede-5600515c4bad running 128 factory-build-2faaef25-31fc-4460-95ab-46a97ed04fd2 running 129 factory-build-f9115156-a0b5-45a2-b623-8a6cc07eaf95 running 132 factory-build-98a70020-c396-4c05-ae50-ed1da0f685f9 running buildhw-aarch64-02.arm.fedoraproject.org | SUCCESS | rc=0 >> Id Name State ---------------------------------------------------- 130 factory-build-2e7c1034-8fdc-433c-bd0c-db6dc08d17bd running 133 factory-build-b2feb358-d0da-4026-b15d-d627a254f223 running 134 factory-build-9130ae49-736a-492a-90bb-65abf97d16ea running 135 factory-build-0b4f76ba-79d0-4a30-86b8-8ac7fe723829 running 139 factory-build-4cc234ca-e4e9-4117-abcf-cbd1efe94530 running 141 factory-build-19c05ac0-900a-4340-b498-d2878093a733 running 142 factory-build-5e4b9412-8c86-4609-bef6-5f3c69c58330 running
We have some issues with vms not being cleaned up. https://github.com/redhat-imaging/imagefactory/issues/397 sets the tone for that.
There is another issue where we are getting
ApplianceError: Image status is FAILED: (35, 'security library: memory allocation failure.')
which has caused this issue to be filed. It has not been fully debugged, we suspect its due to nss, however it is a different error to what we have seen in the past. the issue started happening after the updates being applied to builders on the second of May. The updated package set was:
SDL2-2.0.5-3.fc25.x86_64 Tue 02 May 2017 10:32:31 PM UTC python2-idna-2.5-1.fc25.noarch Tue 02 May 2017 10:32:31 PM UTC libwsman1-2.6.2-8.fc25.x86_64 Tue 02 May 2017 10:32:31 PM UTC json-c-0.12.1-1.fc25.x86_64 Tue 02 May 2017 10:32:31 PM UTC gpgme-1.8.0-10.fc25.x86_64 Tue 02 May 2017 10:32:31 PM UTC fedora-repos-25-3.noarch Tue 02 May 2017 10:32:31 PM UTC emacs-filesystem-25.2-1.fc25.noarch Tue 02 May 2017 10:32:31 PM UTC dmidecode-3.0-7.fc25.x86_64 Tue 02 May 2017 10:32:31 PM UTC bluez-libs-5.44-1.fc25.x86_64 Tue 02 May 2017 10:32:31 PM UTC bash-completion-2.5-1.fc25.noarch Tue 02 May 2017 10:32:31 PM UTC tzdata-2017b-1.fc25.noarch Tue 02 May 2017 10:32:30 PM UTC telnet-0.17-68.fc25.x86_64 Tue 02 May 2017 10:32:30 PM UTC strace-4.16-1.fc25.x86_64 Tue 02 May 2017 10:32:30 PM UTC python2-pyasn1-0.2.3-1.fc25.noarch Tue 02 May 2017 10:32:30 PM UTC python2-boto-2.45.0-3.fc25.noarch Tue 02 May 2017 10:32:30 PM UTC lua-libs-5.3.4-1.fc25.x86_64 Tue 02 May 2017 10:32:30 PM UTC xkeyboard-config-2.20-2.fc25.noarch Tue 02 May 2017 10:32:29 PM UTC screen-4.5.1-1.fc25.x86_64 Tue 02 May 2017 10:32:29 PM UTC pulseaudio-libs-10.0-2.fc25.x86_64 Tue 02 May 2017 10:32:29 PM UTC make-4.1-6.fc25.x86_64 Tue 02 May 2017 10:32:29 PM UTC libgusb-0.2.10-1.fc25.x86_64 Tue 02 May 2017 10:32:29 PM UTC libcroco-0.6.11-3.fc25.x86_64 Tue 02 May 2017 10:32:29 PM UTC iproute-tc-4.10.0-1.fc25.x86_64 Tue 02 May 2017 10:32:28 PM UTC python3-sssdconfig-1.15.2-2.fc25.noarch Tue 02 May 2017 10:32:27 PM UTC python2-dnf-1.1.10-6.fc25.noarch Tue 02 May 2017 10:32:27 PM UTC dnf-plugins-core-0.1.21-5.fc25.noarch Tue 02 May 2017 10:32:27 PM UTC ca-certificates-2017.2.11-1.1.fc25.noarch Tue 02 May 2017 10:32:27 PM UTC zsh-5.2-6.fc25.x86_64 Tue 02 May 2017 10:32:26 PM UTC polkit-0.113-8.fc25.x86_64 Tue 02 May 2017 10:32:26 PM UTC acpid-2.0.28-2.fc25.x86_64 Tue 02 May 2017 10:32:26 PM UTC os-prober-1.74-1.fc25.x86_64 Tue 02 May 2017 10:32:25 PM UTC at-spi2-core-2.22.1-1.fc25.x86_64 Tue 02 May 2017 10:32:25 PM UTC selinux-policy-targeted-3.13.1-225.13.fc25.noarch Tue 02 May 2017 10:32:11 PM UTC policycoreutils-python-utils-2.5-20.fc25.x86_64 Tue 02 May 2017 10:32:09 PM UTC openssh-server-7.4p1-4.fc25.x86_64 Tue 02 May 2017 10:32:09 PM UTC krb5-workstation-1.14.4-7.fc25.x86_64 Tue 02 May 2017 10:32:09 PM UTC glusterfs-fuse-3.10.1-1.fc25.x86_64 Tue 02 May 2017 10:32:09 PM UTC glusterfs-api-3.10.1-1.fc25.x86_64 Tue 02 May 2017 10:32:09 PM UTC dnf-yum-1.1.10-6.fc25.noarch Tue 02 May 2017 10:32:09 PM UTC dnf-automatic-1.1.10-6.fc25.noarch Tue 02 May 2017 10:32:09 PM UTC vim-enhanced-8.0.586-1.fc25.x86_64 Tue 02 May 2017 10:32:08 PM UTC sudo-1.8.19p2-1.fc25.x86_64 Tue 02 May 2017 10:32:08 PM UTC nfs-utils-2.1.1-5.rc2.fc25.x86_64 Tue 02 May 2017 10:32:08 PM UTC NetworkManager-1.4.4-4.fc25.x86_64 Tue 02 May 2017 10:32:08 PM UTC kernel-4.10.13-200.fc25.x86_64 Tue 02 May 2017 10:32:08 PM UTC iptables-services-1.6.0-3.fc25.x86_64 Tue 02 May 2017 10:32:08 PM UTC rsyslog-8.26.0-1.fc25.x86_64 Tue 02 May 2017 10:32:07 PM UTC osinfo-db-20170423-1.fc25.noarch Tue 02 May 2017 10:32:07 PM UTC gtk3-3.22.12-2.fc25.x86_64 Tue 02 May 2017 10:32:07 PM UTC oz-0.15.0-5.fc25.noarch Tue 02 May 2017 10:32:06 PM UTC koji-builder-1.12.0-2.fc25.noarch Tue 02 May 2017 10:32:06 PM UTC euca2ools-3.4.1-1.fc25.noarch Tue 02 May 2017 10:32:06 PM UTC python2-libguestfs-1.36.3-2.fc25.x86_64 Tue 02 May 2017 10:32:05 PM UTC publicsuffix-list-dafsa-20170206-1.fc25.noarch Tue 02 May 2017 10:32:05 PM UTC mock-1.3.4-1.fc25.noarch Tue 02 May 2017 10:32:05 PM UTC libpsl-0.17.0-1.fc25.x86_64 Tue 02 May 2017 10:32:05 PM UTC libguestfs-1.36.3-2.fc25.x86_64 Tue 02 May 2017 10:32:05 PM UTC GeoIP-GeoLite-data-2017.04-1.fc25.noarch Tue 02 May 2017 10:32:05 PM UTC GeoIP-1.6.10-1.fc25.x86_64 Tue 02 May 2017 10:32:05 PM UTC distribution-gpg-keys-1.11-1.fc25.noarch Tue 02 May 2017 10:32:05 PM UTC augeas-libs-1.8.0-1.fc25.x86_64 Tue 02 May 2017 10:32:05 PM UTC python-requestbuilder-0.7.1-1.fc25.noarch Tue 02 May 2017 10:32:04 PM UTC libXi-1.7.9-1.fc25.x86_64 Tue 02 May 2017 10:32:04 PM UTC libnetfilter_conntrack-1.0.6-2.fc25.x86_64 Tue 02 May 2017 10:32:04 PM UTC liblogging-stdlog-1.0.6-1.fc25.x86_64 Tue 02 May 2017 10:32:04 PM UTC libepoxy-1.4.1-1.fc25.x86_64 Tue 02 May 2017 10:32:04 PM UTC iptables-1.6.0-3.fc25.x86_64 Tue 02 May 2017 10:32:04 PM UTC hwdata-0.299-1.fc25.noarch Tue 02 May 2017 10:32:04 PM UTC dosfstools-4.1-1.fc25.x86_64 Tue 02 May 2017 10:32:04 PM UTC kernel-modules-4.10.13-200.fc25.x86_64 Tue 02 May 2017 10:32:00 PM UTC kernel-core-4.10.13-200.fc25.x86_64 Tue 02 May 2017 10:31:59 PM UTC linux-firmware-20170313-72.git695f2d6d.fc25.noarch Tue 02 May 2017 10:31:57 PM UTC mesa-libglapi-13.0.4-3.fc25.x86_64 Tue 02 May 2017 10:31:53 PM UTC mesa-libGL-13.0.4-3.fc25.x86_64 Tue 02 May 2017 10:31:53 PM UTC libglvnd-glx-0.2.999-14.20170308git8e6e102.fc25.x86_64 Tue 02 May 2017 10:31:53 PM UTC vim-common-8.0.586-1.fc25.x86_64 Tue 02 May 2017 10:31:52 PM UTC python-multilib-conf-1.2-1.fc25.noarch Tue 02 May 2017 10:31:52 PM UTC python2-multilib-1.2-1.fc25.noarch Tue 02 May 2017 10:31:52 PM UTC pyOpenSSL-16.2.0-1.fc25.noarch Tue 02 May 2017 10:31:52 PM UTC perl-libs-5.24.1-385.fc25.x86_64 Tue 02 May 2017 10:31:52 PM UTC mesa-libwayland-egl-13.0.4-3.fc25.x86_64 Tue 02 May 2017 10:31:52 PM UTC koji-1.12.0-2.fc25.noarch Tue 02 May 2017 10:31:52 PM UTC vim-minimal-8.0.586-1.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC vim-filesystem-8.0.586-1.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC pango-1.40.5-1.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC openssh-clients-7.4p1-4.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC libnfsidmap-0.27-1.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC json-glib-1.2.6-1.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC git-core-2.9.3-3.fc25.x86_64 Tue 02 May 2017 10:31:50 PM UTC systemd-container-231-14.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC rpcbind-0.2.4-5.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC openssh-7.4p1-4.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC openldap-2.4.44-10.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC nss-tools-3.29.3-1.1.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC nss-sysinit-3.29.3-1.1.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC nss-pem-1.0.3-3.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC nss-3.29.3-1.1.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC NetworkManager-libnm-1.4.4-4.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC mesa-libEGL-13.0.4-3.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC libglvnd-egl-0.2.999-14.20170308git8e6e102.fc25.x86_64 Tue 02 May 2017 10:31:49 PM UTC policycoreutils-python3-2.5-20.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC ntfsprogs-2016.2.22-4.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC mesa-libgbm-13.0.4-3.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC libkadm5-1.14.4-7.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC krb5-libs-1.14.4-7.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC gssproxy-0.7.0-3.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC gnutls-3.5.11-1.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC cups-libs-2.2.0-8.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC audit-libs-python3-2.7.6-1.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC audit-2.7.6-1.fc25.x86_64 Tue 02 May 2017 10:31:48 PM UTC glusterfs-3.10.1-1.fc25.x86_64 Tue 02 May 2017 10:31:47 PM UTC dnf-1.1.10-6.fc25.noarch Tue 02 May 2017 10:31:47 PM UTC crypto-policies-20160921-4.gitf3018dd.fc25.noarch Tue 02 May 2017 10:31:47 PM UTC systemd-udev-231-14.fc25.x86_64 Tue 02 May 2017 10:31:42 PM UTC selinux-policy-3.13.1-225.13.fc25.noarch Tue 02 May 2017 10:31:42 PM UTC policycoreutils-2.5-20.fc25.x86_64 Tue 02 May 2017 10:31:42 PM UTC libdrm-2.4.79-1.fc25.x86_64 Tue 02 May 2017 10:31:42 PM UTC systemd-pam-231-14.fc25.x86_64 Tue 02 May 2017 10:31:41 PM UTC systemd-231-14.fc25.x86_64 Tue 02 May 2017 10:31:41 PM UTC dbus-1.11.12-1.fc25.x86_64 Tue 02 May 2017 10:31:41 PM UTC coreutils-8.25-17.fc25.x86_64 Tue 02 May 2017 10:31:41 PM UTC python3-libs-3.5.3-5.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC python3-dnf-plugins-core-0.1.21-5.fc25.noarch Tue 02 May 2017 10:31:40 PM UTC python3-distro-1.0.3-1.fc25.noarch Tue 02 May 2017 10:31:40 PM UTC python3-3.5.3-5.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC python2-hawkey-0.6.4-3.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC libsemanage-python3-2.5-9.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC gtk-update-icon-cache-3.22.12-2.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC gdk-pixbuf2-modules-2.36.6-1.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC coreutils-common-8.25-17.fc25.x86_64 Tue 02 May 2017 10:31:40 PM UTC system-python-libs-3.5.3-5.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC system-python-3.5.3-5.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC python3-hawkey-0.6.4-3.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC python3-dnf-1.1.10-6.fc25.noarch Tue 02 May 2017 10:31:39 PM UTC libglvnd-0.2.999-14.20170308git8e6e102.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC file-libs-5.29-4.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC file-5.29-4.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC dnf-conf-1.1.10-6.fc25.noarch Tue 02 May 2017 10:31:39 PM UTC cryptsetup-libs-1.7.5-1.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC cryptsetup-1.7.5-1.fc25.x86_64 Tue 02 May 2017 10:31:39 PM UTC polkit-libs-0.113-8.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC p11-kit-trust-0.23.2-3.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC ntfs-3g-2016.2.22-4.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC nss-softokn-freebl-3.29.5-1.0.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC nss-softokn-3.29.5-1.0.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC libsemanage-2.5-9.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC libidn2-2.0.2-1.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC glusterfs-client-xlators-3.10.1-1.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC gdbm-1.13-1.fc25.x86_64 Tue 02 May 2017 10:31:37 PM UTC p11-kit-0.23.2-3.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC nss-util-3.29.5-1.0.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC libsolv-0.6.26-3.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC libseccomp-2.3.2-1.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC iptables-libs-1.6.0-3.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC iproute-4.10.0-1.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC info-6.1-4.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC hawkey-0.6.4-3.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC grep-2.27-2.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC glusterfs-libs-3.10.1-1.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC gdk-pixbuf2-2.36.6-1.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC dbus-libs-1.11.12-1.fc25.x86_64 Tue 02 May 2017 10:31:36 PM UTC systemd-libs-231-14.fc25.x86_64 Tue 02 May 2017 10:31:35 PM UTC glib2-2.50.3-1.fc25.x86_64 Tue 02 May 2017 10:31:35 PM UTC audit-libs-2.7.6-1.fc25.x86_64 Tue 02 May 2017 10:31:35 PM UTC
Metadata Update from @ausil: - Issue assigned to dgregor (was: imcleod)
แบhen there is too many wild running vms we have had image failures due to being unable to allocate memory for the virtual machine. that error message is:
ApplianceError: Image status is FAILED: internal error: process exited while connecting to monitor: 2017-05-06T11:35:28.826779Z qemu-system-arm: cannot set up guest memory 'mach-virt.ram': Cannot allocate memory
as can be seen in task https://koji.fedoraproject.org/koji/taskinfo?taskID=19420033
May 10 12:28:00 buildhw-06.phx2.fedoraproject.org kojid[23494]: 2017-05-10 12:28:00,713 [INFO] {23494} koji.TaskManager:1077 Attempting to take task 19487598 May 10 12:28:03 buildhw-06.phx2.fedoraproject.org kojid[23494]: 2017-05-10 12:28:03,057 [INFO] {23494} koji.TaskManager:678 pids: {19487568: 7113, 19487598: 7114} May 10 12:28:03 buildhw-06.phx2.fedoraproject.org kojid[23494]: 2017-05-10 12:28:03,204 [INFO] {23494} koji.TaskManager:680 open task: {'waiting': None, 'id': 19487598, 'weight': 2.0} May 10 12:28:03 buildhw-06.phx2.fedoraproject.org kojid[23494]: 2017-05-10 12:28:03,204 [INFO] {23494} koji.TaskManager:680 open task: {'waiting': True, 'id': 19487568, 'weight': 1.0} May 10 12:28:06 buildhw-06.phx2.fedoraproject.org kojid[23494]: /usr/lib64/python2.7/optparse.py:1013: PendingDeprecationWarning: The KSOption class is deprecated and will be removed in pykickstart-3. May 10 12:28:06 buildhw-06.phx2.fedoraproject.org kojid[23494]: option = self.option_class(*args, **kwargs) May 10 12:28:06 buildhw-06.phx2.fedoraproject.org kojid[23494]: 2017-05-10 12:28:06,538 [INFO] {7114} koji.build.BaseTaskHandler:3572 dispatching a baseimg builder May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: 2017-05-10 12:28:07,050 [WARNING] {7114} koji.TaskManager:1179 TRACEBACK: Traceback (most recent call last): May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1160, in runTask May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: response = (handler.run(),) May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 158, in run May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: return koji.util.call_with_argcheck(self.handler, self.params, self.opts) May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/lib/python2.7/site-packages/koji/util.py", line 156, in call_with_argcheck May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: return func(*args, **kwargs) May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/sbin/kojid", line 3839, in handler May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: uses this to figure out what plugin to run May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/sbin/kojid", line 3473, in do_images May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: an absolute path to the modified XML May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/sbin/kojid", line 3581, in _buildBase May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: self.logger.info('dispatching %s image builder' % format) May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: File "/usr/sbin/kojid", line 3537, in _checkImageState May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: f_dict['raw'] = False May 10 12:28:07 buildhw-06.phx2.fedoraproject.org kojid[23494]: ApplianceError: Image status is FAILED: (35, 'security library: memory allocation failure.')
the builder traceback is in koji code
I've created a patch for this issue from an upstream commit and tested it locally using mockbuild.
Looks like it did not actually attach... here's an fpaste
hey @breilly, thanks - that should help us find out if the issue is related to oz's initialization of the nss library or not.
someone applied updates to the builders today
python-fedora-0.9.0-3.fc25.noarch Wed 10 May 2017 04:46:57 PM UTC python2-openidc-client-0-3.20170327git5456800.fc25.noarch Wed 10 May 2017 04:46:57 PM UTC kernel-4.10.14-200.fc25.x86_64 Wed 10 May 2017 04:46:57 PM UTC kernel-modules-4.10.14-200.fc25.x86_64 Wed 10 May 2017 04:46:54 PM UTC kernel-core-4.10.14-200.fc25.x86_64 Wed 10 May 2017 04:46:52 PM UTC rsync-3.1.2-3.fc25.x86_64 Wed 10 May 2017 02:34:12 AM UTC python-pycurl-7.43.0-5.fc25.x86_64 Wed 10 May 2017 02:34:12 AM UTC python3-pycurl-7.43.0-5.fc25.x86_64 Wed 10 May 2017 02:34:12 AM UTC publicsuffix-list-dafsa-20170424-1.fc25.noarch Wed 10 May 2017 02:34:12 AM UTC mesa-libwayland-egl-17.0.5-2.fc25.x86_64 Wed 10 May 2017 02:34:12 AM UTC libsolv-0.6.27-1.fc25.x86_64 Wed 10 May 2017 02:34:12 AM UTC libICE-1.0.9-9.fc25.x86_64 Wed 10 May 2017 02:34:12 AM UTC emacs-filesystem-25.2-2.fc25.noarch Wed 10 May 2017 02:34:12 AM UTC ca-certificates-2017.2.14-1.0.fc25.noarch Wed 10 May 2017 02:34:12 AM UTC nss-util-3.30.2-1.0.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC nss-tools-3.30.2-1.0.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC nss-sysinit-3.30.2-1.0.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC nss-softokn-freebl-3.30.2-1.0.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC nss-softokn-3.30.2-1.0.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC nss-3.30.2-1.0.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC nspr-4.14.0-2.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC mesa-libglapi-17.0.5-2.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC mesa-libGL-17.0.5-2.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC mesa-libgbm-17.0.5-2.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC mesa-libEGL-17.0.5-2.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC libX11-common-1.6.5-1.fc25.noarch Wed 10 May 2017 02:34:11 AM UTC libX11-1.6.5-1.fc25.x86_64 Wed 10 May 2017 02:34:11 AM UTC
after restarting kojid it seems things are happy again
https://koji.fedoraproject.org/koji/taskinfo?taskID=19490843
I was talking to @kevin last night about "reverting" to older nss to see if it solved the problem. He noticed that there was newer nss so he suggested we "go forward" and try that first before trying to "go back". so he applied the updates. The image updates from last night failed, though, so we assumed that the updates did not fix the problem.
Are you saying that the new updates did "fix the problem", but that kojid wasn't restarted and that's why the image builds from last night failed?
If so can you kick a .1 f25 atomic host build? the task you linked above is f26 and f26 is broken in other ways unfortunately :(
the .1 compose got further than the memory issue we were seeing, which is good news.
http://kojipkgs.fedoraproject.org/compose/twoweek/Fedora-Atomic-25-20170510.1
Since brendan didn't have commit access to oz (yet) we took the patches and applied them to oz in dist-git http://pkgs.fedoraproject.org/cgit/rpms/oz.git/log/?h=f25. We also built a version that can be tagged into infra when needed: https://koji.fedoraproject.org/koji/buildinfo?buildID=889265
we did not yet install oz into the builders, but dennis did restart kojid after an update that included updating the nss rpms again and we had a successful build last night. We should install the new oz so that we don't have non-deterministic behavior when it comes to initializing the nss library. We can wait until we start seeing the problem again, or we could do it now.
Either way this immediate issue is addressed and this ticket can be closed.
Metadata Update from @mohanboddu: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.