#440 Improve updates process to avoid "windows of doom"

Created 6 years ago by drago01
Modified 6 years ago

= Proposal topic =

A recent PackageKit updated moved the packagekitd binary to a different location and thus breaking the selinux label.

The result is that packagekitd silently fails to start and thus users who updated to this version will no longer receive update notifications and thus potentially render a lot of systems vulnerable as they stop getting updates.

The bug has been fixed in a recent selinux-policy package, but this does not really help the users that updated in this period aka the "window of doom" ... they will just don't get any update notification and presume that they just aren't any.

= Overview =

The package in question has gone through the new updates process but we still failed to detect this major flaw; a breaking the update infrastructure should be avoided at all cost as it dealing with it afterwards is always troublesome and in some cases even "unfixable".

The problem in this specific case it does only happen when selinux is in enforcing mode; which leads to the conclusion that the "proventesters" did test on a non standard configuration (we default to selinux on).

= Problem space =

See above.

= Solution Overview =

There are two problems here; the doomed users that where affected by that update and the fact that we let it happen.

I am afraid the former can't really be fixed the best we can do is to issue some statement and hope that it gets some attention from the press so that at least some of the affected users become aware of the issue.

As for avoiding it from happening again we proventesters should be educated that they should (must) test using some standard default configuration and probably go a step further and demand that they should not only ack an update with "works for me" but provide information on what exactly they tested and on which system configuration that allows the maintainer to judge whether his/her package got enough testing on a wide range of system configuration and seeing what tests where actually made is a way better deciding base than "works for me".

= Active Ingredients =

QA Proventesters, maybe the board / FPL for the "press statement"; package maintainers

It seems that I was mixing two issues here the actual problem that causes update notifications to not appear was https://bugzilla.redhat.com/show_bug.cgi?id=615099 which is even worse as it affects everyone regardless of the status of selinux.

Which still means that we 1) have to deal with it somehow and 2) try to fix up the process to avoid it from happening again in the future.

Sorry for the confusion but unfortunately the problem still remains.

FWIW, I use my system with SELinux enabled, but not enforcing -- which allows me to see any AVCs that are generated with any new packages. In my testing, no AVCs were generated. I have no idea why.

ok, so we wanted at the meeting to do two announcements here:

a) one to the main announce list:

This announcement will let users know that in some cases if they have not seen any updates in a while, they should open a terminal and 'su -c 'yum update'' to get the new PackageKit and gnome-packagekit and selinux policy. Or is there some better way to get them just those updates? 'setenforce 0 / set to permissive mode' and update?
Thoughts?

b) one to the devel-announce / test-announce lists:

This reminds people testing that they should make sure and have selinux enabled and note any avc's that come up in testing.

I'm not sure why there were no AVC's on permissive mode here. ;(

Replying to [comment:6 kevin]:

ok, so we wanted at the meeting to do two announcements here:

a) one to the main announce list:

This announcement will let users know that in some cases if they have not seen any updates in a while, they should open a terminal and 'su -c 'yum update'' to get the new PackageKit and gnome-packagekit and selinux policy. Or is there some better way to get them just those updates? 'setenforce 0 / set to permissive mode' and update?
Thoughts?

There is no new PackageKit needed just gnome-packagekit and selinux-policy.
Set to permissive and update will not work due to the second bug.

So the only options are

1) su -c "yum -y update" // just update everything and reboot
or
2) su -c "yum -y update gnome-packagekit selinux-policy" // only update the affected packages

b) one to the devel-announce / test-announce lists:

This reminds people testing that they should make sure and have selinux enabled and note any avc's that come up in testing.

Yeah that's fine probably add a note to encourage verbose test result i.e not just "works for me"
but "I tested x, y and z and found no regressions".

I'm not sure why there were no AVC's on permissive mode here. ;(

Yeah I talked with Richard about that before filling the ticket we had no clue what was going on there ...

OK, to move this forward we should probably have something like this:

A recent update introduced two nasty bugs in our package management tool (PackageKit) which results into users no longer getting update notifications.

We issued updates with fixes for the above issue but unfortunately, users won't get them unless they update by hand due to missing notifications.

Instructions on how to fix:

1) Open a Terminal (Applications -> Systemtools -> Terminal)
2) Type one of those commands:
su -c "yum -y update"
OR
su -c "yum -y update gnome-packagekit selinux-policy"
3) reboot

We will improve our QA process to avoid such issues from happening again.

Announcement for the provenpackers:


A recent update passed testing despite having a very nasty flaw [1], we should try to avoid this in the future as dealing with such issues are troublesome.

So when testing an update please make sure that you run a close to default configuration (like SELinux in enforcing mode).

As the recent bug has shown permissive isn't enough to catch all issue so please slip enforcing on at least during your testing.

When providing the test result to the maintainer via bodhi please not what you have actually tested and found working / non working.

Verbose information will provide the maintainers a better basis to decide on (push / not push) than simple reports like "works / does not work".


For the first one, the selinux issue means not only are their no notifications, but also no updates possible via PK right? so perhaps:

Several recent updates introduced several bugs in our package update tool (PackageKit) which results into users no longer seeing notifications of new updates or ability to apply them.

We have issued updates with fixes for the above issues but unfortunately, users won't get them unless they apply the pending fixed updates with another tool.

To work around the issue:

1) Open a Terminal (Applications -> Systemtools -> Terminal)
2) Type one of those commands:

su -c "yum -y update" (apply all pending updates now)
OR
su -c "yum -y update gnome-packagekit selinux-policy" (apply updates fixing only this specific bug).

3) reboot your machine, or logout/login (this will fix this issue, but if you applied all updates in the previous step you may want to reboot for other updates that require them).

We are working hard to improve our QA process to avoid such issues from happening again.

The second one looks generally ok to me. I might do some re-wording on it, but the gist is fine. Would you guys like me to send that one out?

Replying to [comment:9 kevin]:

For the first one, the selinux issue means not only are their no notifications, but also no updates possible via PK right? so perhaps:

Several recent updates introduced several bugs in our package update tool (PackageKit) which results into users no longer seeing notifications of new updates or ability to apply them.

We have issued updates with fixes for the above issues but unfortunately, users won't get them unless they apply the pending fixed updates with another tool.

To work around the issue:

1) Open a Terminal (Applications -> Systemtools -> Terminal)
2) Type one of those commands:

su -c "yum -y update" (apply all pending updates now)
OR
su -c "yum -y update gnome-packagekit selinux-policy" (apply updates fixing only this specific bug).

3) reboot your machine, or logout/login (this will fix this issue, but if you applied all updates in the previous step you may want to reboot for other updates that require them).

We are working hard to improve our QA process to avoid such issues from happening again.

The second one looks generally ok to me. I might do some re-wording on it, but the gist is fine. Would you guys like me to send that one out?

Fine by me.

ok, shall I send something out? Or Jared, would you like to do so?

Yes, I'd be happy to send out the update. It will probably be Monday before I get it sent out though, as I'll be travelling most of the day today.

Does this affect Fedora 12 as well as 13?

I talked to Jared just now -- we'll get something out sooner than that. I had some minor wording changes to the text for the announcement.

{{{
Recent updates to Fedora introduced two unexpected bugs that, when combined, result
in users no longer seeing notifications of or being able to apply new updates.
Updates have been shipped to fix these issues, but users will not receive
notifications for them unless they first apply the pending fixed updates using the
"yum" tool.

To work around the issue:

  1. Open a Terminal (Applications > System Tools > Terminal)

  2. Type one of those commands:

a. To choose to apply all pending updates now:
su -c "yum -y update"

b. To choose to apply updates fixing only this specific issue:
su -c "yum -y update gnome-packagekit selinux-policy"

  1. Reboot your machine, or logout and login. Note that logout/login will complete
    the fix for the notification issue. However, if you applied all updates in the
    previous step, you should reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to
our testing and quality assurance processes to avoid this and similar problems in
the future.
}}}

Looks great. One nitpick: "when combined" is not really accurate. Either bug (the selinux one or the gnome-packagekit one) would cause the issue, they don't need to be combined. perhaps "Either of which result in" ?

We still need to make sure this only affects f13 before sending and note that...

Latest draft. Speak now if you have any other changes you'd like to see made before it gets sent out.

{{{
Recent updates to Fedora introduced two unexpected bugs, either of which results in users no longer seeing notifications of or being able to apply new
updates.

Updates to the relevant packages have been shipped to fix these issues, but users will not receive notifications for them unless they first apply the pending fixed updates using the
"yum" tool. (Please note that these issues should only affect Fedora 13. Fedora 12 does not appear to be affected.)

To work around the issue:

  1. Open a Terminal (Applications > System Tools > Terminal)

  2. Type either of the following commands and enter the password for the root user when prompted.
    a. To choose to apply all pending updates now:
    su -c "yum -y update"
    b. To choose to apply updates fixing only the specific issues mentioned above:
    su -c "yum -y update gnome-packagekit selinux-policy"

  3. Log out and and then log back in, or reboot your computer. Note that logout/login will complete the fix for the notification issue. However, if you applied all updates in the previous step, you may need to reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to our testing and quality assurance processes to avoid this and similar problems in the future.

If you have any further questions or require additional assistance, please see https://fedoraproject.org/wiki/Communicating_and_getting_help for ways to get additional help from the Fedora community.
}}}

More slight changes, after continued dialog with stickster:

{{{
Recent updates to Fedora introduced two unexpected bugs, either of which results in users no longer seeing notifications of or being able to apply new
updates.

Updates to the relevant packages have been shipped to fix these issues, but users will not receive notifications for them unless they first apply the pending fixed updates using the
"yum" tool. (Please note that these issues should only affect Fedora 13. Fedora 12 does not appear to be affected.)

To work around the issue:

  1. Open a Terminal (Applications > System Tools > Terminal)

  2. Type either of the following commands and enter the password for the root user when prompted.
    a. To choose to apply all pending updates now:
    su -c "yum -y update"
    b. To choose to apply updates fixing only the specific issues mentioned above:
    su -c "yum -y update gnome-packagekit selinux-policy"

  3. Log out and and then log back in, or reboot your computer. Note that logout/login will complete the fix for the notification issue. However, if you applied all updates in the previous step, you may need to reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to our testing and quality assurance processes to avoid this and similar problems in the future.

If you have any further questions or require additional assistance, please see https://fedoraproject.org/wiki/Communicating_and_getting_help for ways to get additional help from the Fedora community.
}}}

Need to "--skip-broken" to yum commands

{{{
Recent updates to Fedora introduced two unexpected bugs, either of which results in users no longer seeing notifications of or being able to apply new
updates.

Updates to the relevant packages have been shipped to fix these issues, but users will not receive notifications for them unless they first apply the pending fixed updates using the
"yum" tool. (Please note that these issues should only affect Fedora 13. Fedora 12 does not appear to be affected.)

To work around the issue:

  1. Open a Terminal (Applications > System Tools > Terminal)

  2. Type either of the following commands and enter the password for the root user when prompted.
    a. To choose to apply all pending updates now:
    su -c "yum -y --skip-broken update"
    b. To choose to apply updates fixing only the specific issues mentioned above:
    su -c "yum -y --skip-broken update gnome-packagekit selinux-policy"

  3. Log out and and then log back in, or reboot your computer. Note that logout/login will complete the fix for the notification issue. However, if you applied all updates in the previous step, you may need to reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to our testing and quality assurance processes to avoid this and similar problems in the future.

If you have any further questions or require additional assistance, please see https://fedoraproject.org/wiki/Communicating_and_getting_help for ways to get additional help from the Fedora community.
}}}

I talked with Kevin Fenzi a bit, and we agreed that after the parentheses in paragraph 2, we should add something like this:

{{{Packages fixing these issues were pushed to the Fedora software
repositories around 2010-07-22. Users who have done a manual 'yum update'
since that push may have already received these fixes.}}}

We don't want to say "ignore this email," but we should also acknowledge that lots of people do 'yum update' out of habit a couple times a week.

When this notice is out, we'll add a link to the notice to start.fp.o, fp.o, and fp.o/wiki/* to make sure people know there is an issue.

I'll also make sure some of the #fedora admins know so they can set the /topic and spread the word to other admins.

Is there any further actions to take here? Or shall we close this out now?

Doesn't recent !PackageKit try to update itself first? So I think you can still end up with a broken !PackageKit due to an outdated selinux-policy even if you update now, unless the latest !PackageKit has a versioned Conflicts on the old selinux-policy!

Replying to [comment:23 kkofler]:

Doesn't recent !PackageKit try to update itself first? So I think you can still end up with a broken !PackageKit due to an outdated selinux-policy even if you update now, unless the latest !PackageKit has a versioned Conflicts on the old selinux-policy!

UHG!

Yeah you are right lets image this case:
1) Fresh install
2) PackageKit only updates itself, rpm and yum
3) You end up hitting the brokenness.

I am not sure how a conflict would solve that though.
A "Requires: selinux-policy >= fixedVersion" would.

If there are package changes recommended or needed, it would make sense to cc the maintainer here.

Argh, wrong maintainer, sorry for the noise.

Replying to [comment:25 pfrields]:

If there are package changes recommended or needed, it would make sense to cc the maintainer here.

The right place for this is bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=620943

FESCo shouldn't be needed / involved anymore.

I think we can close this now. The process should work for anyone not getting updates, and hopefully we will have PackageKit requiring the needed selinux-policy so new updates will work.

I'm nuking the global message on start.fp.o, fp.o and fp.o/wiki/* today if there are no objections. (It's been 30 days.)

Sounds good to me. Please go ahead and nuke the global message.

Replying to [comment:30 jsmith]:

Sounds good to me. Please go ahead and nuke the global message.

No please don't.

As long as this bug https://bugzilla.redhat.com/show_bug.cgi?id=620943 isn't fixed people doing new installs are STILL affected by the issue.

That bug is now closed and the fixed package pushed out to f13 stable. ;)

Do we need a common bugs entry on this? Or shall we just close it out?

There could be some people still affected, so it would be good to at least have a place to point them for more info when we encounter them.

FWIW, I added text to update the common F13 bugs page:

https://fedoraproject.org/w/index.php?title=Common_F13_bugs&diff=198870&oldid=196791

I think the notice can now be nuked.

I agree. Ian? Can you remove the note? and then we can close this...

Login to comment on this ticket.