Building x11vnc for EPEL8 repeatedly failed on buildvm-ppc64le-10.iad2.fedoraproject.org with:
Traceback (most recent call last): File "/usr/lib/python3.13/site-packages/koji/daemon.py", line 1421, in runTask response = (handler.run(),) ~~~~~~~~~~~^^ File "/usr/lib/python3.13/site-packages/koji/tasks.py", line 341, in run self.createWorkdir() ~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.13/site-packages/koji/tasks.py", line 367, in createWorkdir os.makedirs(self.workdir) ~~~~~~~~~~~^^^^^^^^^^^^^^ File "<frozen os>", line 227, in makedirs OSError: [Errno 30] Read-only file system: '/var/tmp/koji/tasks/2330/132212330'
https://koji.fedoraproject.org/koji/taskinfo?taskID=132211577 https://koji.fedoraproject.org/koji/taskinfo?taskID=132212236
I've disabled that builder along with buildvm-s390x-08.s390.fedoraproject.org (reported on matrix) until we can look at them later.
Metadata Update from @kevin: - Issue assigned to kevin
Turns out buildvm-s390x-08 was fine.
buildvm-ppc64le-10 wasn't letting me login at all. I rebooted it and it came up fine.
Looking in remote logs:
May 2 11:17:22 buildvm-ppc64le-10 kernel: ------------[ cut here ]------------ May 2 11:17:22 buildvm-ppc64le-10 kernel: WARNING: CPU: 4 PID: 2351242 at fs/btrfs/extent_map.c :422 unpin_extent_cache+0x78/0x170 May 2 11:17:22 buildvm-ppc64le-10 kernel: Modules linked in: nft_reject_ipv6 nft_reject nft_cha in_nat nf_nat nft_ct poly1305_generic chacha_generic chacha20poly1305 macsec cfg80211 pppoe pppo x ppp_generic slhc tun xsk_diag vsock_diag sctp_diag can_bcm sctp ip6_udp_tunnel udp_tunnel blue tooth can_j1939 can_isotp can_raw can crypto_user overlay ip_set_hash_ip ip_set ib_core nfsv3 nf s_acl nfs lockd grace nfs_localio netfs bonding tls rfkill ip6t_REJECT nf_reject_ipv6 xt_conntra ck nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables binfmt_misc xfs joydev virtio _net net_failover virtio_balloon failover vmx_crypto auth_rpcgss sunrpc loop nfnetlink zram vsoc k_loopback lz4hc_compress vmw_vsock_virtio_transport_common vsock lz4_compress crct10dif_vpmsum crc32c_vpmsum virtio_blk bochs pseries_wdt i6300esb fuse virtio_console aes_gcm_p10_crypto crypt o_simd cryptd May 2 11:17:22 buildvm-ppc64le-10 kernel: CPU: 4 UID: 0 PID: 2351242 Comm: kworker/u165:14 Tain ted: G L 6.13.8-200.fc41.ppc64le #1 May 2 11:17:22 buildvm-ppc64le-10 kernel: Tainted: [L]=SOFTLOCKUP May 2 11:17:22 buildvm-ppc64le-10 kernel: Hardware name: IBM pSeries (emulated by qemu) POWER9 (architected) 0x4e1202 0xf000005 of:SLOF,HEAD hv:linux,kvm pSeries May 2 11:17:22 buildvm-ppc64le-10 kernel: Workqueue: btrfs-endio-write btrfs_work_helper May 2 11:17:22 buildvm-ppc64le-10 kernel: REGS: c00000030b9e3960 TRAP: 0700 Tainted: G L (6.13.8-200.fc41.ppc64le) May 2 11:17:22 buildvm-ppc64le-10 kernel: MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI, LE> CR: 24000244 XER: 2004008c May 2 11:17:22 buildvm-ppc64le-10 kernel: CFAR: c000000000a26a60 IRQMASK: 0 #012GPR00: c0000000 00a2766c c00000030b9e3c00 c0000000024daa00 0000000000000000 #012GPR04: 0000000000018000 0000000000008000 0000000000000001 0000000000000000 #012GPR08: 00000000000000ff 0000000000000001 0000000000010000 0000000000000000 #012GPR12: 0000000000000000 c0000004ffff9280 c000000000287ef8 c00000000af17580 #012GPR16: 0000000000000000 c00000001e3c7000 c0000002aacc8538 0000000000009000 #012GPR20: 0000000000008000 c00000001f1f7400 0000000000004044 0000000000018000 #012GPR24: 0000000000000000 000000000001ffff c00000001e3c7000 0000000000008000 #012GPR28: 00000000000562ea 0000000000018000 c0000002aacc8508 0000000000000000 May 2 11:17:22 buildvm-ppc64le-10 kernel: NIP [c000000000a27678] unpin_extent_cache+0x78/0x170 May 2 11:17:22 buildvm-ppc64le-10 kernel: LR [c000000000a2766c] unpin_extent_cache+0x6c/0x170 May 2 11:17:22 buildvm-ppc64le-10 kernel: Call Trace: May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3c00] [c000000000a2766c] unpin_extent_cache+0x6c/0x170 (unreliable) May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3c90] [c000000000a0c5c4] btrfs_finish_one_ordered+0x434/0xd90 May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3de0] [c000000000a33170] finish_ordered_fn+0x20/0x40 May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3e00] [c000000000a561f4] btrfs_work_helper+0x134/0x270 May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3e40] [c000000000279678] process_one_work+0x1f8/0x520 May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3ef0] [c00000000027b08c] worker_thread+0x33c/0x4f0 May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3f90] [c000000000288020] kthread+0x130/0x140 May 2 11:17:22 buildvm-ppc64le-10 kernel: [c00000030b9e3fe0] [c00000000000ded8] start_kernel_thread+0x14/0x18 May 2 11:17:22 buildvm-ppc64le-10 kernel: Code: e93e0000 eb490208 48daf879 60000000 38c00001 7fe3fb78 7f65db78 7fa4eb78 4bfff211 7c690074 7c7f1b78 7929d182 <0b090000> 2c230000 41820080 e8e30018 May 2 11:17:22 buildvm-ppc64le-10 kernel: ---[ end trace 0000000000000000 ]--- May 2 11:17:23 buildvm-ppc64le-10 kernel: BTRFS warning (device vda3): no extent map found for inode 1452013466 (root 256) when unpinning extent range [98304, 131072), generation 353002 May 2 11:17:23 buildvm-ppc64le-10 kernel: ------------[ cut here ]------------ May 2 11:17:23 buildvm-ppc64le-10 kernel: BTRFS: Transaction aborted (error -2) May 2 11:17:23 buildvm-ppc64le-10 kernel: WARNING: CPU: 4 PID: 2351242 at fs/btrfs/inode.c:3227 btrfs_finish_one_ordered+0xbd4/0xd90 May 2 11:17:23 buildvm-ppc64le-10 kernel: Modules linked in: nft_reject_ipv6 nft_reject nft_chain_nat nf_nat nft_ct poly1305_generic chacha_generic chacha20poly1305 macsec cfg80211 pppoe pppox ppp_generic slhc tun xsk_diag vsock_diag sctp_diag can_bcm sctp ip6_udp_tunnel udp_tunnel bluetooth can_j1939 can_isotp can_raw can crypto_user overlay ip_set_hash_ip ip_set ib_core nfsv3 nfs_acl nfs lockd grace nfs_localio netfs bonding tls rfkill ip6t_REJECT nf_reject_ipv6 xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables binfmt_misc xfs joydev virtio_net net_failover virtio_balloon failover vmx_crypto auth_rpcgss sunrpc loop nfnetlink zram vsock_loopback lz4hc_compress vmw_vsock_virtio_transport_common vsock lz4_compress crct10dif_vpmsum crc32c_vpmsum virtio_blk bochs pseries_wdt i6300esb fuse virtio_console aes_gcm_p10_crypto crypto_simd cryptd May 2 11:17:23 buildvm-ppc64le-10 kernel: CPU: 4 UID: 0 PID: 2351242 Comm: kworker/u165:14 Tainted: G W L 6.13.8-200.fc41.ppc64le #1 May 2 11:17:23 buildvm-ppc64le-10 kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP May 2 11:17:23 buildvm-ppc64le-10 kernel: Hardware name: IBM pSeries (emulated by qemu) POWER9 (architected) 0x4e1202 0xf000005 of:SLOF,HEAD hv:linux,kvm pSeries May 2 11:17:23 buildvm-ppc64le-10 kernel: Workqueue: btrfs-endio-write btrfs_work_helper May 2 11:17:23 buildvm-ppc64le-10 kernel: REGS: c00000030b9e39f0 TRAP: 0700 Tainted: G W L (6.13.8-200.fc41.ppc64le) May 2 11:17:23 buildvm-ppc64le-10 kernel: MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000242 XER: 0000008c May 2 11:17:23 buildvm-ppc64le-10 kernel: CFAR: c00000000024560c IRQMASK: 0 #012GPR00: c000000000a0cd60 c00000030b9e3c90 c0000000024daa00 0000000000000025 #012GPR04: 000000010001ecbe 000000010001ecbe c00000030b9e39e8 00000004f9fe0000 #012GPR08: 0000000000000027 0000000000000000 0000000000000000 0000000000000000 #012GPR12: 2036363433313032 c0000004ffff9280 c000000000287ef8 c00000000af17580 #012GPR16: 0000000000000000 c00000001e3c7000 c0000002aacc8538 0000000000009000 #012GPR20: 0000000000008000 c00000001f1f7400 0000000000004044 0000000000018000 #012GPR24: 0000000000000000 000000000001ffff 0000000000000000 c0000002aacc8508 #012GPR28: 0000000000000000 c000000005a9f6c0 fffffffffffffffe c0000000145d28a0 May 2 11:17:23 buildvm-ppc64le-10 kernel: NIP [c000000000a0cd64] btrfs_finish_one_ordered+0xbd4/0xd90 May 2 11:17:23 buildvm-ppc64le-10 kernel: LR [c000000000a0cd60] btrfs_finish_one_ordered+0xbd0/0xd90 May 2 11:17:23 buildvm-ppc64le-10 kernel: Call Trace: May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3c90] [c000000000a0cd60] btrfs_finish_one_ordered+0xbd0/0xd90 (unreliable) May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3de0] [c000000000a33170] finish_ordered_fn+0x20/0x40 May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3e00] [c000000000a561f4] btrfs_work_helper+0x134/0x270 May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3e40] [c000000000279678] process_one_work+0x1f8/0x520 May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3ef0] [c00000000027b08c] worker_thread+0x33c/0x4f0 May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3f90] [c000000000288020] kthread+0x130/0x140 May 2 11:17:23 buildvm-ppc64le-10 kernel: [c00000030b9e3fe0] [c00000000000ded8] start_kernel_thread+0x14/0x18 May 2 11:17:23 buildvm-ppc64le-10 kernel: Code: 91210060 4bfc9849 60000000 2c230000 81210060 91210060 418200d0 3c62ff85 7fc4f378 3863f658 4b8387cd 60000000 <0fe00000> 81210060 3b800001 91210060 May 2 11:17:23 buildvm-ppc64le-10 kernel: ---[ end trace 0000000000000000 ]--- May 2 11:17:23 buildvm-ppc64le-10 kernel: BTRFS: error (device vda3 state A) in btrfs_finish_one_ordered:3227: errno=-2 No such entry May 2 11:17:23 buildvm-ppc64le-10 kernel: BTRFS info (device vda3 state EA): forced readonly
kernel is 6.13.8-200.fc41
I'll keep an eye out for this happening every again.
Metadata Update from @kevin: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.