#2207 Questions about the 7 day waiting period in Bodhi
Opened a month ago by bex. Modified 6 days ago

As we improve our gating in Rawhide, I'd like to ask some questions and have us give some thought to the Bodhi process.

  • Do we know what percentage of updates for shipping releases receive karma versus don't?
  • Do we know what percentage of updates have to wait out the full 7 days versus receive sufficient karma to move faster?

The current process for updates is effectively 9 or so days by the time you wait on testing, wait out the 7 day bodhi karma period and then wait for a push to the updates repo. Is this long time period needed? Statistics will tell us, I think. One idea might be to offer a reduced period for packages that have gating tests in Rawhide, and later in the releases.


If somebody has those numbers, it's bodhi. I suggest to ask Fedora Infra to fetch them.

One idea might be to offer a reduced period for packages that have gating tests in Rawhide, and later in the releases.

I'm all in favor of approving individually approved reduced testing time for packages with good enough tests. However nothing automated. If the package is gated on rpmlint or similar, we should not reduce the testing time. I've seen packages with always passing %check far more often than I think is healthy. I don't have illusions about this getting any different in gating.

OTOH I think that packagers that have useful and good enough tests also don't have trouble getting karma if they need to speed things up.

What I don't understand is: What's wrong with the wait time? if the update fixes some bugs that bother users, the impacted can karma it up to have them faster (or at least fetch it from testing to fix their problem). If the update is just "meh" kind of update, it can wait a week (+ up to 2 extra days).

I'm all in favor of approving individually approved reduced testing time for packages with good enough tests. However nothing automated. If the package is gated on rpmlint or similar, we should not reduce the testing time. I've seen packages with always passing %check far more often than I think is healthy. I don't have illusions about this getting any different in gating.

Agreed 100%. If we make gating tests easy enough, there are few excuses.

OTOH I think that packagers that have useful and good enough tests also don't have trouble getting karma if they need to speed things up.
What I don't understand is: What's wrong with the wait time? if the update fixes some bugs that bother users, the impacted can karma it up to have them faster (or at least fetch it from testing to fix their problem). If the update is just "meh" kind of update, it can wait a week (+ up to 2 extra days).

I disagree here. As an example, I recently package gocryptfs. It is a new piece of software for Fedora (and relatively new in the universe). To do it I needed to package several layers of dependencies. Review times plus 9 days for each level was a long delay on landing the package. As there are no other consumers of the dependencies any karma I could beg for would probably be fake.

Now that it is packaged it isn't used much. Therefore I am again in a fake karma begging situation. There is a pending update that fixes a dependency issue that wasn't caught in the original packaging. I can think of two people who might give it karma at the moment. I suspect it is likely to get none as this just isn't a huge draw.

However, it feels wrong to make me wait out an arbitrary timer just because the package isn't heavily used. It's not like anything will change in 9 days ...

To do it I needed to package several layers of dependencies. Review times plus 9 days for each level was a long delay on landing the package. As there are no other consumers of the dependencies any karma I could beg for would probably be fake.

That is a matter of workflow. I recommend using buildroot overrides (those happen +- right away) when needed and only submit one update with the entire stack of deps + the app.

Now that it is packaged it isn't used much. Therefore I am again in a fake karma begging situation. There is a pending update that fixes a dependency issue that wasn't caught in the original packaging. I can think of two people who might give it karma at the moment. I suspect it is likely to get none as this just isn't a huge draw.

If nobody uses the package but you, why the rush to push it sooner?

However, it feels wrong to make me wait out an arbitrary timer just because the package isn't heavily used. It's not like anything will change in 9 days ...

What are you waiting for exactly?

I still think we should really allow manual pushes to stable at any time. The package maintainers should know better when their package is stable than some testers with nothing more than a FAS account. And waiting a week with no feedback does not make the package any more stable either.

But if the waiting period remains, I would propose changing the Bodhi rules to allow stable pushes 7 days after update submission rather than 7 days after the push to testing actually happens. That would make things much more predictable for maintainers and not hold them accountable for delays in infrastructure that are none of their fault. (And it would actually give an incentive to rel-eng for getting testing pushes out in a timely manner, since any delay would chew on the testing time. ;-) )

That said, I still believe (as I already did when this policy was first introduced) that the waiting period is entirely unnecessary and just causes more issues. (Regressions often make it through anyway, and then the fix for the regression gets unnecessarily delayed. Back in the day, I and some other maintainers would just send regression fixes directly to stable, and many users would not even notice the regression because it would have been fixed already in the next push one day later. The current policies prevent that.)

Caveat: We are discussing a set of specifics so this may not generalize

To do it I needed to package several layers of dependencies. Review times plus 9 days for each level was a long delay on landing the package. As there are no other consumers of the dependencies any karma I could beg for would probably be fake.

That is a matter of workflow. I recommend using buildroot overrides (those happen +- right away) when needed and only submit one update with the entire stack of deps + the app.

That only works if reviewers will allow it. I got push back that I couldn't get reviews completed until dependencies were packaged. If this is a workflow issue, then we need better docs to clear this up. IIRC, buildroot overrides are only lightly documented themselves.

Now that it is packaged it isn't used much. Therefore I am again in a fake karma begging situation. There is a pending update that fixes a dependency issue that wasn't caught in the original packaging. I can think of two people who might give it karma at the moment. I suspect it is likely to get none as this just isn't a huge draw.

If nobody uses the package but you, why the rush to push it sooner?

Because I am trying to build usage. In my case I wrote an opensource.com article and specifically cited the easy of installation in Fedora. Sure, 9 more days doesn't matter, but it feels arbitrary.

However, it feels wrong to make me wait out an arbitrary timer just because the package isn't heavily used. It's not like anything will change in 9 days ...

What are you waiting for exactly?

I'd like my package in Fedora. As a contributor this important to me. It is the payoff for all the effort. I'd like my updates in Fedora for the same reason. I don't mind waiting if the time buys us something collectively good. But an untested package sitting for 9 days instead of something less is just an arbitrary road block.

I suggest we wait for bodhi stats before we debate this further. We may discover that my example package is a minority of updates that doesn't get karma and that the vast majority of packages flow through the system super fast and super tested.

I got push back that I couldn't get reviews completed until dependencies were packaged

Reviews should happen on rawhide. Did you get pushback for not providing updates in stable Fedoras? That seems wrong.

Anyway I agree that the documentation is not the best. And waiting for stats is fine as well with me. Would you mind requesting them from the Bodhi people?

I have done dozens of updates on my packages and never, ever received enough karma for a stable push. In fact, I receive karma so rarely it's not even worth mentioning. I have been a packager for a few years now and I work closely with upstream to contribute code, resolve bugs, make suggestions, communicate the needs of the distro, etc.

What I observe in terms of this issue is that I am pushing updates when the cadence of changes is accelerating. I find that I can be on the tail end of that 7-9 day waiting period just as something new comes through and I need to start all over. Frustrating.

I went to look at the data in bodhi's database ( from yesterday db dumps https://infrastructure.fedoraproject.org/infra/db-dumps/bodhi2.dump.xz)

For Fedora 30
Total stable updates: 4652
Total stable updates with +3 karma: 566
Total stable updates with karma that did not reach +3: 1079
Total stable updates with no karma: 3007

So that is around 12% of update that get enough karma to be pushed to stable based on karma (I cannot really tell when the karma threshold was reached)

For Fedora 29
Total stable updates: 7849
Total stable updates with +3 karma: 940
Total stable updates with karma that did not reach +3: 1718
Total stable updates with no karma: 5191

So again roughly 12% of update reaching the +3 karma threshold.

Indeed, the feedback system sorta works for a few high-profile packages (though those often get positive karma very quickly with insufficient testing, leading to regressions slipping through), but does not work at all for the vast majority of packages in Fedora. I maintain several packages that rarely get any karma at all, e.g. calamares (an alternative installer), kdelibs3 (a compatibility library), etc.

I think the karma integer is not a reliable way to measure the quality of an update at all (heavily-used packages get too much too quickly, rarely-used packages get too little too slowly), and the decision should be put back in the hands of the maintainer. I also think that autokarma is inherently dangerous and should not be allowed. Promoting an update to stable should always be a concious decision. And maintainers who, for whatever reason (hopefully a good one!) do not care what testing, if any, their package get, should just be allowed to push their package directly to stable.

I think direct stable pushes (or stable pushes after only 1-2 days of testing if the maintainer wishes so; I think both should be allowed) make a lot of sense in at least 4 cases: 1. a fix for a regression from a previous update, 2. an urgent bugfix (e.g., a critical security or data loss fix), 3. a niche package that nobody is going to test anyway, or 4. a new package with no Obsoletes.

And I would even consider removing the +1/0/-1 radiobuttons entirely, requiring testers to fill in the freeform text with something reasonable and the maintainers to actually read the text and base their decisions on the details and not just on an arbitrary integer.

I think the karma integer is not a reliable way to measure the quality of an update at all (heavily-used packages get too much too quickly, rarely-used packages get too little too slowly), and the decision should be put back in the hands of the maintainer.

That's why we need to emphasize that the default limit of 3 is just a limit and encourage high profile packages to use higher limit and the other way around.

Promoting an update to stable should always be a concious decision. And maintainers who, for whatever reason (hopefully a good one!) do not care what testing, if any, their package get, should just be allowed to push their package directly to stable.

Regardless if we decide that the 7 day limit needs to be changed, I strongly suggest that having at least some kind of period when anybody can say: "Stop that train!" even when the packager is 100% sure that their update can go directly to stable is absolutely necessary. People make mistakes. I certainly do.

I think direct stable pushes (or stable pushes after only 1-2 days of testing if the maintainer wishes so; I think both should be allowed) make a lot of sense in at least 4 cases: 1. a fix for a regression from a previous update, 2. an urgent bugfix (e.g., a critical security or data loss fix), 3. a niche package that nobody is going to test anyway, or 4. a new package with no Obsoletes.

  1. If the previous regression is terrible, ask for karma by the people who reported it. if only you have noticed it, only after the update was pushed to stable (maybe it was pushed to fast?) you can always reach at devel.

  2. I believe we have a way to do this. Just ask releng. I am totally sure they can push an update direct in case of a terrible disaster kind of thing.

  3. I don't quite see the urgency to push that fast.

  4. No obsoletes, no extra provides, no explicit conflicts, no file conflicts, no subpackages that overlap with existing (sub)packages, no supplements, no suggests, and maybe I have forgot something. One can argue that the reviewer should check those - and yes, the reviewer can provide the +1 karma if needed as well.

I think the karma integer is not a reliable way to measure the quality of an update at all (heavily-used packages get too much too quickly, rarely-used packages get too little too slowly), and the decision should be put back in the hands of the maintainer.

That's why we need to emphasize that the default limit of 3 is just a limit and encourage high profile packages to use higher limit and the other way around.

What would you suggest for specific guidance on the karma integer? I feel like we should try to be algorithmic about it and if possible get it implemented as the suggested default in bodhi. To figure this out it might be interesting to do an analysis of which packages actually get karma at all and what they got.

Regardless if we decide that the 7 day limit needs to be changed, I strongly suggest that having at least some kind of period when anybody can say: "Stop that train!" even when the packager is 100% sure that their update can go directly to stable is absolutely necessary. People make mistakes. I certainly do.

+100 - however, I think the time period should be dependent on the kind of package it is. I suggest three categories and some day counts below. Note the days should be summed:

  1. Package part of the default installation of any edition: +4 days (possibly with notification to the edition working group)
  2. Package identified as high impact by FESCo (manual decision): +3 days (ideally with notification to devel list or something similar)
  3. Package is a dependency (non-leaf packages) +3 days (ideally with notification to the packagers who depend on it directly via email)
  4. Any package with no other days already: +3 days

This should result in 3 days to potentially 10 days (assuming FESCo declared a default install package to also be high impact). Add in the notifications and we may actually get packages looked at.

I think direct stable pushes (or stable pushes after only 1-2 days of testing if the maintainer wishes so; I think both should be allowed) make a lot of sense in at least 4 cases: 1. a fix for a regression from a previous update, 2. an urgent bugfix (e.g., a critical security or data loss fix), 3. a niche package that nobody is going to test anyway, or 4. a new package with no Obsoletes.

  1. If the previous regression is terrible, ask for karma by the people who reported it. if only you have noticed it, only after the update was pushed to stable (maybe it was pushed to fast?) you can always reach at devel.

I agree except for the appeal to devel. That feels like karma begging which we should try to avoid, imho.

  1. I believe we have a way to do this. Just ask releng. I am totally sure they can push an update direct in case of a terrible disaster kind of thing.

Yes. This should also be the super-exception, not the rule.

  1. I don't quite see the urgency to push that fast.

Just because a package has no testers doesn't mean it doesn't have users.

  1. No obsoletes, no extra provides, no explicit conflicts, no file conflicts, no subpackages that overlap with existing (sub)packages, no supplements, no suggests, and maybe I have forgot something. One can argue that the reviewer should check those - and yes, the reviewer can provide the +1 karma if needed as well.

I feel like my days strategy and any proposed karma count suggestions should fix this case.

What would you suggest for specific guidance on the karma integer? I feel like we should try to be algorithmic about it and if possible get it implemented as the suggested default in bodhi. To figure this out it might be interesting to do an analysis of which packages actually get karma at all and what they got.

I base my karma limits on previous experience. So yes, having a default of 1 for new packages and a "self learned" default for others would be awesome. Note that this needs to take a lot of things into account:

  • is there a bugzilla assigned? how many?
  • is the fedora branched (e.g. F31 soon)? stable newest (e.g. F30 now)? or stable older (e.g. F29 now)?
  • mutlipackage updates?
  • what kind of update it is with what severity?
  • is the person who gives the most karma to this package also the person who submitted the update?

I think that it might end up being more work than we can allocate on this. I'd rather have engineering focus on https://pagure.io/fesco/issue/2115

Package identified as high impact by FESCo (manual decision)

JFYI critpath packages already have higher waiting limits.

That feels like karma begging which we should try to avoid, imho.

I think there are legitimate use case for karma begging. E.g. compare: "I have this bugfix that prevents Fedora from eating your pet cat, add karma now to help me push it or more kittens will die." with "Libfoo now added support for the FGHJK protocol when fetching data packats over a modem connection, I need karma to push this ASAP"

Just because a package has no testers doesn't mean it doesn't have users.

And we should thrive to make this better, not just lower the time barrier.

@bex:

What would you suggest for specific guidance on the karma integer? I feel like we should try to be algorithmic about it and if possible get it implemented as the suggested default in bodhi. To figure this out it might be interesting to do an analysis of which packages actually get karma at all and what they got.

The issue we have is packages not getting any karma at all, so just setting a lower positive limit (i.e., 1) does not actually do anything to solve this issue. (In fact, I already set stablekarma to 1 for all my updates. I disable autopushing anyway, so if I deem the feedback insufficient, I'll just wait longer before hitting "Push to stable", I don't need Bodhi to enforce that.)

I see only 2 ways to improve the situation for those packages (the ones that never get any karma): allow maintainers to push updates with 0 karma to stable sooner, or abolish karma altogether.

however, I think the time period should be dependent on the kind of package it is. I suggest three categories and some day counts below. Note the days should be summed:

Package part of the default installation of any edition: +4 days (possibly with notification to the edition working group)
Package identified as high impact by FESCo (manual decision): +3 days (ideally with notification to devel list or something similar)
Package is a dependency (non-leaf packages) +3 days (ideally with notification to the packagers who depend on it directly via email)
Any package with no other days already: +3 days

This should result in 3 days to potentially 10 days (assuming FESCo declared a default install package to also be high impact). Add in the notifications and we may actually get packages looked at.

I think this is overly complicated. I am also surprised that you are suggesting delays up to 10 days after complaining that 7 days are too long (though to be fair, the current maximum wait time is 14 days with critpath). If all you want is to have a "stop the presses" grace period, 1 or 2 days should be sufficient.

And I believe that the decision should in the end be made by the maintainer, applying common sense, rather than by software following an algorithm that gets more and more complicated to work around shortcomings of the previous heuristic, without ever being able to truly judge the quality of the update (because for that, it would need to understand the freeform comments, also look at Bugzilla and potentially even upstream communication channels, etc.). Humans are still more intelligent than computers.

Just because a package has no testers doesn't mean it doesn't have users.

Yes. But what is the point of waiting with a stable push for those packages, when you can be reasonably certain that nobody is going to test them anyway? It's just a waste of time.

I feel like my days strategy and any proposed karma count suggestions should fix this [new package] case.

Not really. Your proposed days strategy would decrease the pointless wait time for a new package (that per definition is not going to break existing installations) from 7 days to 3 days, but those are still 3 days too many. Your proposed karma count suggestions would not change anything because it is already possible to set the stablekarma to 1, and allowing to set it to 0 would be equivalent to abolishing the waiting time and hence conflict with your proposed days stratgegy.

If all you want is to have a "stop the presses" grace period, 1 or 2 days should be sufficient.

That is usually not enough to sync the repo mirrors.

@churchyard:

I base my karma limits on previous experience. So yes, having a default of 1 for new packages and a "self learned" default for others would be awesome.

I think a human can do this much more reliably than a heuristic algorithm. As you point out, maintainers already base the threshold on their experience with the package. What is wrong with that?

Though I still believe that being asked an integer threshold in advance is not the optimal experience and that we should instead do manual pushes based on a judgment of the situation at the time of the stable push, including the non-numeric feedback. (This is currently possible to the extent allowed by the policies by setting stablekarma to 1 and disabling autokarma, but I think it should always work that way and not require me to set the same options again and again.)

I think there are legitimate use case for karma begging. E.g. compare: "I have this bugfix that prevents Fedora from eating your pet cat, add karma now to help me push it or more kittens will die." with "Libfoo now added support for the FGHJK protocol when fetching data packats over a modem connection, I need karma to push this ASAP"

This makes sense, but I still think allowing direct stable pushes would be a better solution for these cases, as it would save the communication overhead.

Just because a package has no testers doesn't mean it doesn't have users.

And we should thrive to make this better, not just lower the time barrier.

This has been repeated ever since the waiting time was introduced. The situation has not changed. It is unlikely to ever change. You cannot force volunteers to do things. We have to find a solution that works with the testing (or lack thereof) we actually get.

Unrealistic testing requirements are exactly why Fedora Legacy failed, and Fedora proper has not only repeated the same mistakes, but is still not willing to fix the issue even after running into the same problems a second time.

If all you want is to have a "stop the presses" grace period, 1 or 2 days should be sufficient.

That is usually not enough to sync the repo mirrors.

In my experience, the mirrors I get are normally not more than 1 or (rarely) 2 days out of date. I think mirrors with a lagging time above 2 days are entirely unacceptable, because it also means security fixes will be delayed by that amount of time, and should get kicked out of the dynamic mirror lists by MirrorManager.

Make the (karma, days-to-wait) a function of update severity: unspecified/low = (3 karma, 6 days); medium = (1 karma, 4 days); high = (0 karma, 2 days).

I also like to think that FESCo tickets are the place where we rubber stamp decisions, not invent them entirely. Should this be moved to the devel mailing list instead?

I would also like to push some updates faster, however I don't know any reasonable criteria which should apply to all packages.

+1 to move discussion to devel@

While the current karma thresholds & waiting period duration can centrainly be adjusted, I personally think there are bigger issues preventing the Bodhi community QA process from working correctly.

The main issue is the long waiting periods & times when the update is locked & can't be manipulated during the pushes. This effectvely makes the process needlessly asynchronous, unpredictable and time consuming.

In an ideal scenario testrs would be able to get testing and award karma right after the update has been created, once enough karma is reached the package would be pushes to stable in say 3 days. This should give enough time to slam on the breaks if a critical issue is found by one of the testers.

Also if a serious bug is found, it should be possible to just upload a new build and start testing again, starting from 0 karma & reset timeout. The only thing lost is potentially a fraction of the 3 day timeout.

Unfortunatelly, the current process is far from this flexibility, in my opinion severely limitting peoples wilingness to test package updates. You can't simply fire an update and notify interested parties to get testing, but repetedly wait for arbitrary long times and only then tell people to test & start from scratch if any serious bug is actually found:

  • it takes at least a day for package to reach testing, much longer if the push fails for some reason
  • people can test the package even before it reaches testing, but they (AFAIK) have to fetch the RPMs manually, which is cumbersome
  • during the push the update is locked, so even if the CI attached to Bodhi or early user testing is telling yout the build is wrong, you need to wait for the push to finish before you can attach a new build
  • once the package finally reaches testing, you (or some automated mechanism) can finally tell people to test
  • users can finally properly test it and give it karma
  • the problem is, that if some serious issues are actually found, you start from scratch, adding another couple of days to the process as the package slowly travels back to the testing repository
  • once it reaches the repository, testers can be again notified
  • then if the testers are still interested in testing the update, they might not find an more issues and the package migh then finally reach the stable repository another couple days later

Basically, the only delay in this whole process should be explicit safety timeout (specified to 3 days ain the example above) with all the other steps happening in real time (say in minutes at most). That would, in my opinion, make the whole process much less cumbersome for all parties involved, would make it much more likely package updates are properly tested and any issues promptly fixed before another round of testing & release to stable once a reasoanble timeout is reached.

people can test the package even before it reaches testing, but they (AFAIK) have to fetch the RPMs manually, which is cumbersome

And can sometimes hide upgrade path issues.

+1 to anything @m4rtink said

I agree that the infrastructure delays are an issue (though I don't think it is the main issue, the biggest delays still come from the policies). Unfortunately, it looks like the mirroring infrastructure is the main bottleneck, and much of it is in the hand of third parties.

Ideally, what should happen once an update gets submitted for testing is that it immediately gets automatically signed and moved to the Koji tag for testing, then pushed (by Bodhi or Koji) to the master mirror, and then pushed from there to all the other mirrors through a push mirroring setup. (The metadata refresh issue should be solved now with the introduction of zchunk metadata. In principle, zchunk should even be usable to update the metadata on the mirrors.) This should all be doable in a matter of minutes.

There is at least one Free Software project using push mirroring (ClamAV). But I have been told our mirror admins (or their superiors) do not like the idea at all and several of them would discontinue mirroring Fedora if this became a requirement. The mirrors insist on doing scheduled pulls and are reluctant to even pull more frequently. So we are stuck with at most daily pushes and with unbounded extra lag until the content trickles down to the mirrors (also taking in account that they are hierarchically tiered, adding extra delays).

But do we actually need updates testing to be on the mirrors ? According to the current stats, only about 1500 packages got any karma (and thus hopefully testing) at all so far during the F30 cycle. Let's say removing the delays makes people 3x times more likely to test (which is pretty ambitious).

That would mean the master mirrors would have to serve 4500 packages with the load spread over 5 months (since F30 Bodhi activation time). That sounds totally doable to me & could avoid lot of churn for the mirrors if they hosted only the less changing stable repo content.

Actually, going by the stats, 3000 packages have likely not been touched by anyone during their time in F30 Bodhi so far - all these packages have been pushed to mirrors, only to be dumped again a bit later without anyone making use of them.

As an example of urgent security fixes that I believe should go directly to stable, see the 6 updates for CVE-2019-14744 listed in:
https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org/message/TYV3NJSVQBJRME2VKCGO4CQGJQYSQ6TS/

Only 1 of them (kf5-kconfig for F30) got the necessary karma to go to stable so far. And this is how it usually works (or rather, fails to work): updates for Fedora n-1 get no karma anymore, updates for compatibility libraries never get any karma.

@kkofler I'm glad the maintainer didn't have an option to push this update directly to stable (or sooner after e.g. 3 days). Is this what you are advocating for? After all, it is a new package.

In general, I think I agree with @kkofler here: the integer karma is not a good measure of actual the amount of testing. It's useless on packages with small tester communities (88% of all updates didn't reach their karma threshold) and also useless on those packages with large communities like Firefox which regularly blasts past its autokarma value despite carrying regressions (such as in NSS).

I'd like to propose another option though, now that we have gating I think the rule should be as follows:

"For any update consisting of packages which have at least one additional gating test beyond any mandated in Fedora by default, we permit the maintainer to push to stable at their discretion so long as they have not received negative karma."

To explain further: I agree with Kevin that our package maintainers should be trusted to understand their packages' needs better than anyone else. I also agree that we want to try to avoid pushing bad updates whenever possible. So I think the middle-ground here is to say that they're allowed more control over pushing if they have added automated tests for gating. Will these tests always be exhaustive? No, of course not. Any at all is still better than none. And if we found people submitting fake or useless tests to bypass karma rules, we can disable them and/or take punitive actions against the maintainer.

So, if a package receives any negative karma (the "emergency stop" button), the discretionary push is disabled and it reverts to the current behavior until a new build is added. Which means that it can still be pushed at the end of the 7 day period, even over the noted objections. (This is meant to work around those cases where people file negative karma for updates that don't fix a bug they didn't claim to fix and similar situations.)

I think we should remove "so long as they have not received negative karma", because maintainers are already at discretion to ignore negative karma. Negative karma disables autopush, and that is enough. So if either gating tests fail, or negative karma is submitted, manual maintainer override is always required. I think this is necessary and sufficient.

+1 to @sgallagh's proposal with this one change.

IIUC, this would require changes to bodhi to allow setting "stable days" to anything >= 0. This is already the case in rawhide, so it should not be too much trouble to allow it in other branches too.

I think we should remove "so long as they have not received negative karma", because maintainers are already at discretion to ignore negative karma. Negative karma disables autopush, and that is enough. So if either gating tests fail, or negative karma is submitted, manual maintainer override is always required. I think this is necessary and sufficient.

Well, my thought was that this would force them to wait through the timeout before ignoring the feedback, but I can be convinced to drop that section.

"For any update consisting of packages which have at least one additional gating test beyond any mandated in Fedora by default, we permit the maintainer to push to stable at their discretion so long as they have not received negative karma."

I'm not really happy with this due to:

  1. Most of the new packages don't have/need any gating test
  2. Tests are not really reliable, still
  3. Nothing prohibits pushing test which would just run exit 0

I think we need to define some better criteria than just "one additional gating test".

"For any update consisting of packages which have at least one additional gating test beyond any mandated in Fedora by default, we permit the maintainer to push to stable at their discretion so long as they have not received negative karma."

I'm not really happy with this due to:

Most of the new packages don't have/need any gating test

I don't think new packages should bypass the wait time. See the example in https://pagure.io/fesco/issue/2207#comment-589333

Tests are not really reliable, still

Then the worst-case scenario is the current state. What's the problem? If the tests aren't passing, then they don't get to short-circuit.

Nothing prohibits pushing test which would just run exit 0

I addressed that in the part you didn't copy: "[I]f we found people submitting fake or useless tests to bypass karma rules, we can disable them and/or take punitive actions against the maintainer."

I think we need to define some better criteria than just "one additional gating test".

So, this is already a long ticket and I cannot possibly answer/address everything, but:

  • I am pretty against any changes to this policy that don't get discussed on devel list. There are a lot of people who have strong opinions on this, and only a few of them are here in this ticket and we need to take their needs into account as well.

  • Just because an update is made, pushed to testing, gets 0 karma and pushes to stable after a week doesn't mean that it had no testing or testers. I suspect there's a small group of people that provide karma, then a much larger group that enables updates-testing and ignores it unless they find some problem, then they provide -1 karma. I'm not sure how to detect this group, but saying there's no testing is... not accurate. We can't know.

  • We have no push ability to mirrors. They pull from us as they wish. We could ask them for some way to push, but I suspect most would not be interested in allowing that as it means they have to run some kind of special service on their mirrors. Many of them mirror lots of projects, not just fedora, and want them all to just be pulled by rsync.

  • We could stop mirroring testing I suppose, but it's unclear how many people are actually using it. Hint: it's going to be a lot more than bother to add karma.

  • Perhaps we should hear from QA here? @adamwill or @tflink or @kparal might have some opinions. ;)

We might also look at the longer term here? How do we want this to look after we have CI in place and tests on everything?

This is a very long thread and I haven't read everything, but...

  • I think there needs to be some recognition we've already loosened things here substantially over time. We now essentially default to all updates that don't receive any feedback automatically going stable after 7 days. I wouldn't call that a conservative policy. Does it frustrate me when I have my own little special package that I just know is perfect because I'm perfect but it didn't get any karma so I have to wait 7 days? Sure. Is it good when someone sends out something completely broken? Well, yeah. Particularly note the 'time-based autopush' feature is new and was specifically implemented in response to the 'lots of updates don't get karma' point. We've only had that feature for, what, a couple of months? It seems pretty early to be pushing for loosening the rules yet again.

  • I also agree that improving stuff like locking and the time it takes to push updates would help a lot. If we could queue changes during a locking period that would be great (I appreciate it may be difficult to implement though). It would also be nice if auto-obsoletion of existing updates was removed or made optional, to help with the situation @pkfed identified - where you want to send out more changes, but doing so forces you to obsolete or edit your existing update, perhaps after 5 or 6 days...

Sorry, one more: I also have concerns about the 'let people push gated packages immediately' proposal. I just don't see that we have any confidence that any particular gating tests for any particular packages prove very hard that the package isn't going to break anything else. Just saying "OK, if you've got any optional gating at all you can push whatever you want any time!" seems pretty...bold to me.

Could we pretty please, move the discussion to devel?

OK, one more: we could also benefit from a comprehensive check that what Bodhi currently implements actually lines up with our updates policy, because there's certainly been drift there, and I don't know how much of it has been fixed. For instance, on the 'days to stable' front - the policy says between Bodhi enablement point and Beta release, non-critpath packages should only have to wait 3 days if they don't reach the maintainer's chosen karma threshold; it's 7 days from Beta release onwards...but critpath packages are supposed to have to wait 14 days, all the time, if they don't get sufficient karma.

On karma, the policy says that critpath updates need +1 from Bodhi enablement to Beta, and +2 from Beta release onwards. The non-critpath policy has sorta been changed in response to Bodhi changes, which is a bit odd. Originally the rule was non-critpath packages needed +1 to be pushed before the time threshold was up. The +3-by-default automatic push was just a convenience feature in Bodhi; once your update got to +1 you could push it manually. Now it seems like these things have been sorta smooshed together in the Bodhi webui: the webui won't let you push updates manually if they've reached +1 (or +2 for crithpath updates after Beta) but haven't reached whatever autopush threshold you set, and the policy says "reach the prescribed karma level for that update", which reads as if it's talking about the Bodhi autopush karma threshold...BUT you can still push an update that's reached +1 or +2 but has not reached its autopush threshold using the bodhi CLI. This is kind of a mess.

@churchyard:

@kkofler I'm glad the maintainer didn't have an option to push this update directly to stable (or sooner after e.g. 3 days). Is this what you are advocating for? After all, it is a new package.

It is/was not a "new package with no Obsoletes", as I had written for a reason. It is obvious that Obsoletes can break things badly, especially abusive Obsoletes as here. This could be caught by an automated check.

To prevent this kind of abuse by inexperienced packagers, it would also be possible to limit direct stable pushes to packagers with a minimum amount of experience or to provenpackagers, instead of banning it completely even for those who know what they are doing (as we currently do).

Or we need to find a way to better supervise new sponsorees. That update with that Obsoletes should really not have been filed at all, not even to testing. In fact, it should not even have been built for Rawhide.

So, there are 3 possible solutions (automated check for bad Obsoletes, direct stable pushes only for experienced packagers, and/or better onboarding process) that would all be more suited to the problem you point out than what we have now.

@adamwill:

We now essentially default to all updates that don't receive any feedback automatically going stable after 7 days. I wouldn't call that a conservative policy.
Particularly note the 'time-based autopush' feature is new and was specifically implemented in response to the 'lots of updates don't get karma' point.

The thing is, I don't think autopush is a useful feature at all. Either I am sure that the update is good to go when I file it, then I don't want it to wait 7 days before eventually going out automatically without anything having changed anyway, or I am not so sure, then I definitely want to have another look at it before hitting the "Push to stable" button. And I think maintainers should have another look before promoting an update to stable.

I also disagree that this is a loosening of policy because it only automates what the policy already allowed. So this is purely an implementation change, not a policy change. (But I do think that the implementation is dangerous because it eliminates the human check.)

but critpath packages are supposed to have to wait 14 days, all the time, if they don't get sufficient karma.

Ouch! Enforcing that even in the branched period is going to make the branched period even more painful.

The non-critpath policy has sorta been changed in response to Bodhi changes, which is a bit odd. Originally the rule was non-critpath packages needed +1 to be pushed before the time threshold was up. The +3-by-default automatic push was just a convenience feature in Bodhi; once your update got to +1 you could push it manually. Now it seems like these things have been sorta smooshed together in the Bodhi webui: the webui won't let you push updates manually if they've reached +1 (or +2 for crithpath updates after Beta) but haven't reached whatever autopush threshold you set, and the policy says "reach the prescribed karma level for that update", which reads as if it's talking about the Bodhi autopush karma threshold...BUT you can still push an update that's reached +1 or +2 but has not reached its autopush threshold using the bodhi CLI. This is kind of a mess.

The Bodhi web UI allows you to set the stable karma to 1 and to disable autopush based on karma. (I always use that setting.) This is equivalent to what the policy allows. It is just not the default for some reason.

What I do not understand is what the point of having a stable karma setting for manual pushes is to begin with. Why can it not always be set to 1? Nothing forces me to actually push this update at +1 when autopush is disabled, so it is my decision as a maintainer anyway (and it really depends on the update and on the freeform feedback I get whether I will actually push it at +1 or wait for more feedback), and I am allowed (both by the policy and by the code) to set the threshold to 1, so why would I ever want the threshold set to something different? Higher stable karma thresholds are only meaningful for autokarma.

See also the discussion in: https://github.com/fedora-infra/bodhi/issues/772#issuecomment-249533602 – the bugs were fixed, thankfully, but my questions as to why this threshold setting is there at all were ignored, sadly.

"The Bodhi web UI allows you to set the stable karma to 1 and to disable autopush based on karma. (I always use that setting.) This is equivalent to what the policy allows. It is just not the default for some reason."

Oh, right, I forgot you could do this. It just seems like an odd mechanism to have the ability to set your own manual push threshold; what's the use case for setting it to 5 but disabling autopush?

"The thing is, I don't think autopush is a useful feature at all. Either I am sure that the update is good to go when I file it, then I don't want it to wait 7 days before eventually going out automatically without anything having changed anyway, or I am not so sure, then I definitely want to have another look at it before hitting the "Push to stable" button."

That's not my experience. I am usually "pretty sure" my updates are going to be fine, but there's always a chance there could be a problem I missed. That's what the 7 days are for. If someone flags up a problem within those 7 days, I go ahead and disable autopush (if they didn't submit negative karma) and look into it. Otherwise, the update goes out without me having to remember to go back in and push it manually.

It just seems like an odd mechanism to have the ability to set your own manual push threshold; what's the use case for setting it to 5 but disabling autopush?

That's exactly the point I tried to make in https://github.com/fedora-infra/bodhi/issues/772#issuecomment-249533602 (in words) and https://github.com/fedora-infra/bodhi/issues/772#issuecomment-249534729 (mathematically). Unfortunately, neither explanation has convinced the Bodhi team that the setting is redundant. Maybe you can explain it better to them? (To me as a mathematician, it is so obvious that it is hard to explain.)

Metadata Update from @zbyszek:
- Issue tagged with: meeting

a month ago

We discussed this doing todays' FESCo meeting (2019-08-19). We decided that this merits a discussion on fedora-devel before any FESCo-level decisions.
ACTION: ignatenkobrain to talk with bexelbie about starting a discussion on fedora-devel.

Metadata Update from @zbyszek:
- Issue untagged with: meeting

a month ago

Metadata Update from @ignatenkobrain:
- Issue assigned to ignatenkobrain

a month ago

Any chance of getting the discussion started soon?

Metadata Update from @churchyard:
- Issue tagged with: stalled

10 days ago

I have talked with @bex few days ago iand I have some ideas how we could improve this. I'll write up thoughts tomorrow.

Please bring your thoughts to the devel mailing list, not this ticket. Thank you.

Login to comment on this ticket.

Metadata