Testcloud's user sessions have historically had a window of a race condition. That's the case due to unavailability of a mechanism to know for sure if the cloud-init has finished its job guest side.

Guest image bringup from this context would look like:

• spawn qemu VM
• ssh starts early
• race window opens
• cloud-init applies our amendments to the ssh, users, etc.
• ssh reads the changed values
• race window closes

The symptoms of a race window include a never-ending connection waiting, asking for password even with passed ssh key, unavailability of predefined users and so on.

Evaluated options were:

• Change the default ssh port from the guest perspective in the cloud-init's runcmd, and setup mapping for the different port, so unchanged ssh port is invisible for the tmt < this seems like the perfect candidate

• Let cloud-init asynchronously spawn something on some arbitrarily set port, which the tmt would wait for to be open before attempting to do anything

• Send a http request from the guest from the cloud-init to the host (I don't like this one, it seems like we'll get a different set of bugs due to user networking from guest to host fun)

• Leverage virtiofs - tmt part isn't ready yet, user sessions aren't supported by libvirt at all for this

• Leverage the direct kernel boot, append sshd mask to the /proc/cmdline, which results in ssh being stopped until cloud-init spawns it (described in the issue, has its own problems too)

With the second one being the favorite solution due to SElinux interfering with ssh port changing and unavailability of semanage in guest images.