#11 btrfs subvolume creation on a loop-mounted filesystem fails when using qemu-user
Closed: Fixed 3 years ago by ngompa. Opened 3 years ago by dcavalca.

This came up while working on https://pagure.io/appliance-tools/pull-request/15. When doing cross-arch builds with mock, it uses qemu-user-static under the hood, and that seems to create some weirdness around loop devices that confuses the btrfs tools. Here's a repro, to be run on an x86_64 fc33 or rawhide system:

$ sudo dnf install mock qemu-user-static wget
$ sudo usermod -a -G mock $USER
$ newgrp mock
$ mock --root fedora-rawhide-armhfp --install btrfs-progs util-linux
$ mock --root fedora-rawhide-armhfp --chroot 'rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && losetup /dev/loop9 foo.img &&  mkfs.btrfs /dev/loop9 && mkdir /foo && mount /dev/loop9 /foo && btrfs subvol create /foo/subvol && umount /foo && losetup -d /dev/loop9'

The output of the last command will be:

INFO: Unable to build arch armv7hl natively on arch x86_64. Setting forcearch to use software emulation.
INFO: mock.py version 2.4 starting (python version = 3.9.0)...
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start(bootstrap): chroot init
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled package manager cache
Start(bootstrap): cleaning package manager metadata
Finish(bootstrap): cleaning package manager metadata
INFO: enabled HW Info plugin
Mock Version: 2.4
INFO: Mock Version: 2.4
Finish(bootstrap): chroot init
Start: chroot init
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled package manager cache
Start: cleaning package manager metadata
Finish: cleaning package manager metadata
INFO: enabled HW Info plugin
Mock Version: 2.4
INFO: Mock Version: 2.4
Finish: chroot init
INFO: Running in chroot: ['rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && losetup /dev/loop9 foo.img &&  mkfs.btrfs /dev/loop9 && mkdir /foo && mount /dev/loop9 /foo && btrfs subvol create /foo/subvol && umount /foo && losetup -d /dev/loop9']
Start: chroot ['rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && losetup /dev/loop9 foo.img &&  mkfs.btrfs /dev/loop9 && mkdir /foo && mount /dev/loop9 /foo && btrfs subvol create /foo/subvol && umount /foo && losetup -d /dev/loop9']
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.575975 s, 1.9 GB/s
ERROR: device scan failed on '/dev/loop9': Function not implemented
btrfs-progs v5.7 
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Label:              (null)
UUID:               40fa6fa2-b0df-4274-847f-22c5021aa98b
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         single            8.00MiB
  System:           single            4.00MiB
SSD detected:       yes
Incompat features:  extref, skinny-metadata
Runtime features:   
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/loop9

ERROR: cannot create subvolume: Function not implemented
Create subvolume '/foo/subvol'
Finish: chroot ['rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && losetup /dev/loop9 foo.img &&  mkfs.btrfs /dev/loop9 && mkdir -p /foo && mount /dev/loop9 /foo && btrfs subvol create /foo/subvol && umount /foo && losetup -d /dev/loop9']

Note the "ERROR: device scan failed on '/dev/loop9': Function not implemented" from the mkfs, and the "ERROR: cannot create subvolume: Function not implemented" from the subvolume creation. Note that the mkfs error repros only when using a loop device, but the subvolume error repros with an image as well (which makes sense, as I'm fairly sure under the hood it's just using a loop device anyway):

$ mock --root fedora-rawhide-armhfp --chroot 'rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && mkfs.btrfs -f foo.img && mkdir -p /foo && mount -o loop foo.img /foo && btrfs subvol create /foo/subvol && umount /foo'
INFO: Unable to build arch armv7hl natively on arch x86_64. Setting forcearch to use software emulation.
INFO: mock.py version 2.4 starting (python version = 3.9.0)...
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start(bootstrap): chroot init
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled package manager cache
Start(bootstrap): cleaning package manager metadata
Finish(bootstrap): cleaning package manager metadata
INFO: enabled HW Info plugin
Mock Version: 2.4
INFO: Mock Version: 2.4
Finish(bootstrap): chroot init
Start: chroot init
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled package manager cache
Start: cleaning package manager metadata
Finish: cleaning package manager metadata
INFO: enabled HW Info plugin
Mock Version: 2.4
INFO: Mock Version: 2.4
Finish: chroot init
INFO: Running in chroot: ['rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && mkfs.btrfs -f foo.img && mkdir -p /foo && mount -o loop foo.img /foo && btrfs subvol create /foo/subvol && umount /foo']
Start: chroot ['rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && mkfs.btrfs -f foo.img && mkdir -p /foo && mount -o loop foo.img /foo && btrfs subvol create /foo/subvol && umount /foo']
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.622159 s, 1.7 GB/s
btrfs-progs v5.7 
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               ef40f9a8-0037-4179-a072-2fc77180393a
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              51.19MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Runtime features:   
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  foo.img

ERROR: cannot create subvolume: Function not implemented
Create subvolume '/foo/subvol'
Finish: chroot ['rm -f foo.img && dd if=/dev/zero of=foo.img bs=1G count=1 && mkfs.btrfs -f foo.img && mkdir -p /foo && mount -o loop foo.img /foo && btrfs subvol create /foo/subvol && umount /foo']
$ 

Metadata Update from @ngompa:
- Issue tagged with: Dev, Kernel, Utils

3 years ago

@josef Could you take a look at this? It seems like it would make sense for btrfs-progs to be able to create filesystems and subvolumes even under emulation.

Metadata Update from @ngompa:
- Issue assigned to josef

3 years ago

qemu-user requires manual work to add support for any system call that Linux exposes. While very good, it absolutely does not have 100% coverage of all Linux system calls, especially the 1000's of ioctls.

There is a patch series from a few days ago to add support for many btrfs ioctls

https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg05594.html

I'd assume that without these ioctls support in qemu-user most btrfs related operations are likely to fail when run in the emulated environment.

When I run the above mock commands, it prints the missing ioctls to stderr.

We first get

Unsupported ioctl: cmd=0x5331
Unsupported ioctl: cmd=0x5331
Unsupported ioctl: cmd=0x5331
Unsupported ioctl: cmd=0x50009404
ERROR: device scan failed on '/dev/loop9': Function not implemented

0x5331 is BTRFS_IOC_GET_FSLABEL, aka FS_IOC_GETFSLABEL

0x50009404 is BTRFS_IOC_SCAN_DEV

Later we get more errors

Unsupported ioctl: cmd=0x5331
Unsupported ioctl: cmd=0x5000940e
ERROR: cannot create subvolume: Function not implemented

0x5000940e is BTRFS_IOC_SUBVOL_CREATE

All three of the ioctls seen here are indeed missing. 2 of the 3 missing ioctls are implemented by the patch series I pointed to previously, but FS_IOC_GETFSLABEL is not implemented. More missing ioctls might appear once those first ones are fixed.

@berrange So we just have to wait for this to be fixed in QEMU?

Metadata Update from @ngompa:
- Assignee reset

3 years ago

Metadata Update from @ngompa:
- Issue untagged with: Kernel

3 years ago

Testing a scratch build, the patches on qemu-devel appear to fix the problems reported above. Once the patches are accepted by the QEMU maintainers, It'd be possible to cherry pick them into the Fedora QEMU binaries for F33/F34 at least. Please file a BZ against QEMU in Fedora if that's desired.

Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed

3 years ago

Login to comment on this ticket.

Metadata
Boards 1
Development Status: Done