FESCo ticket for the Feature: https://fedorahosted.org/fesco/ticket/690

Note that at the end there's some changes that haven't been added to the Feature page. In particular, it seems like the submitter should be able to implement the symlinking as a %pretrans script in the filesystem package (either in pure lua or that's invoked by anaconda and preupgrade with the lua script in filesystem merely telling the user that they need to invoke the script manually if they're doing a yum upgrade).

I've asked that the Feature page be updated but there's been no response -- One possible way forward if the page is not updated before FPC looks at it would be that the FPC update the page with all of our required changes and our understanding of the proposed changes as seen in the FESCo ticket. Then FESCo could have the task of verifying with the Feature owner that the feature is still worthwhile in their view.

We are updating the feature page right now. Stay tuned.

done

"Packages MUST NOT place files or directories in the /bin, /sbin, /lib or /lib64 directories. Instead, the /usr/bin, /usr/sbin, /usr/lib, and /usr/lib64 directories must be used. Packages must assume that /bin, /sbin, /lib, and /lib64 are symbolic links to the /usr/bin, /usr/sbin, /usr/lib, and /usr/lib64 directories, respectively."

This wording was approved (+1:7, 0:0, -1:0), however, we are holding on making it effective until:

• The upgrade script is reviewed and approved by the FPC
• An explanation is provided to the FPC to show how yum upgrades will be handled.
• Testing needs to be done to show that the upgrade will work

FWIW I talked with Harald and Kay about this and promised to comment here as well:

I STRONGLY recommend against using %pretrans in filesystem (or any other package for that matter) to do the upgrade mass-move / link-conversion. It's much safer to use a separate script to do it instead (like they originally planned). Various points in no particular order:

• Should anything go wrong in the conversion, an external script can stop, leaving at least SOME hope of recovering the situation. A %pretrans scriptlet cannot stop the transaction, so no matter what might go wrong in there, rpm will stampede through the ruins, most likely making even bigger (much bigger) mess in the process.

• %pretrans and test-transactions (which yum likes to do) don't go well together: the test-transaction cannot execute scripts, so it sees the system as it is now (seeing conflicts and other problems, or not). Whereas the real transaction happens after having moved things around, which might cause new conflicts or other issues, but at that point its too late to back out as the conversion has already occurred. With an external script that is run first, both the test-transaction and the real transaction will see the system the same way, eliminating such issues.

• %pretrans is an extremely unfriendly place to do anything at all, considering the limited facilities of Lua (or work around it by tricks such as forking an external shell to run content embedded in the %pretrans script, which the test-version of filesystem %pretrans I saw was doing). The whole %pretrans thing is a scary hack that's best seen as a last resort to do the minimal required tweak that just cannot be done elsewhere, not something you want to use to make massive filesystem reorganizations. An external script can be written to be much more robust and cleaner than anything in %pretrans ever could because of the "hostile" and limited environment it runs in.

• Current versions of rpm and yum interact (because of the test-transaction that yum does but rpm doesn't) in such a way that using %pretrans for this will only not do what it was meant to do, in addition it can end up eating files that were supposed to be installed. This is something to be investigated and fixed regardless of what happens with the usrmove feature, but perhaps serves as an example of what fragile and dangerous hack %pretrans is.

The FPC considers that the criteria has been met and the hold is lifted. (+1:6, 0:0, -1:1)

Reading through the meeting minutes... there's a hole in the plan:

15:39:25 <haraldh> the filesystem rpm still has the lua check
15:39:37 <haraldh> and fails to install on an unconverted system
15:39:48 <spot> yes, but thats the problem
15:39:53 <haraldh> and all critical packages will have "Conflicts: filesystem < 3"
15:40:06 <tibbs|h> The lua check doesn't work.
15:40:10 <haraldh> it does
15:40:15 <tibbs|h> pretrans can't abort the transaction.
15:40:25 <haraldh> tibbs|h, correct.. but %pre can
15:40:30 <haraldh> and it does

%pre can NOT abort the transaction any more than %pretrans, no scriptlet can. What %pre will abort is the install/upgrade of /that/ package, not the entire transaction. The transaction will continue on its merry way despite such a failure, and despite ending up with broken dependencies - rpm does not track dependencies inside a transaction in progress.

Okay, that's a really big problem. We want the transaction to stop if the filesystem isn't prepared.

Panu, James pointed out that if RPM was able to handle changes between directories and symlinks, this issue might not be a problem. Any chance we will see that?

Beyond some special cases like replacing an empty directory with a symlink (which I suppose could be reasonably implemented but wouldn't help this at all), on a generic level the dir <-> symlink change is an incredibly complicated and hard problem.

When the directory is not empty, you'd have to do something about the contents in order to change it to a symlink. So rpm would need to copy stuff around, possibly crossing filesystem boundaries in which case it'd have to track and recreate any hardlinks involved, and take this move into account when calculating disk-space needs. If stuff is moved to a new target, it's possible for the target directory to already have files/directories of the same name (but different content): one could be a binary and the other one a directory, or a symlink to the file we are trying to move on top of it. Or an admin-created symlink to the directory we're about to replace, or a symlink to some entirely different directory with some extra content in it. What if there are two packages trying to replace the same directory with a different symlink? Etc etc etc... that's just a couple issues off the top of my head, I'm sure there are many many more. All this would have to be precalculated to be able to abort on conflicts and other problems before making any changes to the system.

In other words, its a pink pony, and a particularly shiny bright one at that.

A special-purpose script which "knows" what can be generally expected to be in the directories involved can perhaps take some shortcuts and cross fingers, but even then I consider this whole thing an incredibly dangerous thing to attempt. The only sane and safe way to accomplish the new /usr world order is to require fresh installs. Really.

If this is going to proceed anyway, I suppose we could hack up and carry a fedora-specific patch in rpm to prevent any bigger disasters from trying to upgrade non-converted system. One possibility might be runtime detecting whether /bin etc are symlinks and faking up an rpmlib() provide on a "converted" system, and have the filesystem package require that, so trying to upgrade a non-converted system would abort early with unmet dependencies.

This is placed on hold again, pending Panu's acceptance of a code change to rpmlib to prevent yum updates from succeeding unless rpm is upgraded and the filesystem is migrated via dracut.
(+1:5, 0:0, -1:0)

The patch by Harald & Kay implements exactly what I suggested in my previous comment in a fairly non-intrusive way, I can live with that for a couple of releases. Doesn't mean I'm happy with the upgrade scenario, but with the conversion done outside the live system (ie dracut/anaconda/preupgrade) and this rpm-level guard we've at least tried to minimize the casualties.

Announce Text:

As part of the /usrmove feature in Fedora 17, Fedora packages MUST NOT place files or directories in the /bin, /sbin, /lib or /lib64 directories. Instead, the /usr/bin, /usr/sbin, /usr/lib, and /usr/lib64 directories must be used. Packages must assume that /bin, /sbin, /lib, and /lib64 are symbolic links to the /usr/bin, /usr/sbin, /usr/lib, and /usr/lib64 directories, respectively.

This is effective immediately for new packages, however, packagers are not required to implement this change for distributions older than Fedora 17.

https://fedoraproject.org/wiki/Packaging:Guidelines#Filesystem_Layout