#1890 updating the FTBFS cleanup policy
Closed 2 years ago Opened 2 years ago by zbyszek.

This is a continuation of #1877: large number of packages FTBFS in F28.

Status quo:
Policy documents:
- https://fedoraproject.org/wiki/Fails_to_build_from_source
- https://docs.pagure.org/releng/sop_deprecate_ftbfs_packages.html
Both of those contain some outdated comments, but the gist is that

Somebody runs a regular rebuild
New bugs for each failing package will be filed in Bugzilla
Packages with unresolved FTBFS bugs at Feature Freeze for the N+2 release will be removed from the distribution.

That policy is reasonable, so I propose to most update/clarify it:

  1. explicitly say that the automatic creation creation of FTBFS bugs will happen after each mass rebuild

  2. remove all the outdated mentions of Matt Domsch and monthly rebuilds, BugZappers, CVS, PackageDB, etc.

  3. replace "Packages with unresolved FTBFS bugs at Feature Freeze for the N+2 release will be removed" with "Packages that failed to build in two consecutive mass rebuilds will be removed 6 weeks after the second mass rebuild". This actually gives a similar timeline, but Feature Freeze is not part of the process anymore. 6 weeks is after the branch point and bodhi activation, which is not ideal, but I think it's necessary to give maintainers some time to fix those bugs.

  4. Introduce a rule that changing the FTBFS bug status to ASSIGNED/anything-else-that-is-not-NEW means that the bug is being worked on.

I we start enforcing this policy, i.e. file bugs regularly after each mass rebuild, and actually retire packages, packages that FTBFS will go away after approximately 7½ months.

On the rel-eng side, I'd like to propose the following requirements:
- file FTBFS bugs for all packages which fail in each mass rebuild
- attach build logs to those bugs (root.log, build.log, mock.log)
- exclude any packages from retirement where the FTBFS bug is not in state NEW and there has been a change in the bug state in the last month. (This gives maintainers a way to automatically postpone the retirement.)
- actually do the retirements around 6 weeks after the FTBFS
- if packages are retired, close the FTBFS bugs with an explanation as CLOSED/WONTFIX.

If wanted, we could add the retirement process to the schedule.


My gut tells me this isn't aggressive enough, but my brain tells me that perfect is the enemy of good.

+1

I'd make one amendment: Any bug that is not updated to ASSIGNED/foo six weeks after the first failed mass rebuild will automatically be flagged for the non-responsive maintainer process. That way, we have a full Fedora cycle to find a new maintainer to pick it up before the auto-retirement occurs.

Also, starting the non-responsive maintainer process can light a fire under the maintainer if they are in fact still around.

It that's their only package, that'd make sense. But if they have more than one package, then I'm not sure.

For point #3, I wonder if we should only retire the package on Rawhide and not on the branched release? I could see arguments either way, but I would worry it could be disruptive to the branched release if we did it that late in the cycle. What do others think?

i'm +1 to @sgallagh's suggestion, though I think we should perhaps take it case-by-case in light of @zbyszek's comment.

If we only do master, not the newly-branched version, the time from initial detection of the problem until package retirement will be ~13½ months. That seems way too much.

So maybe we should be more aggressive instead, and retire packages two weeks after the second mass rebuild failure? This is still ~6½ months from the initial bug report, but would give the rest of the distro more time to handle the fallout.

It that's their only package, that'd make sense. But if they have more than one package, then I'm not sure.

I don't see this as being any different than the cases where a non-responsive maintainer issue is forwarded to us manually. We get plenty of those where the packager isn't maintaining one package but has others. The only difference is that this case could be automated.

The primary difference would be in outcome; we might remove the maintainer from just this package, rather than all of their packages.

We get plenty of those where the packager isn't maintaining one package but has others.

OK. I just don't want us to automatically remove all the packages from the maintainer.

@zbyszek I agree - 13 months is too long. I'd +1 to 2 weeks after the second mass rebuild - after all, they already had ~6 months and didn't fix it in that time, so 2 more weeks is a very reasonable last chance to fix it imo. I think this would address my concern too.

I'd make one amendment: Any bug that is not updated to ASSIGNED/foo six weeks after the first failed mass rebuild will automatically be flagged for the non-responsive maintainer process. That way, we have a full Fedora cycle to find a new maintainer to pick it up before the auto-retirement occurs.

The conclusion is not quite right, as soon as the process is fixed, orphaned packages will be retired again when they are orphaned for six or more weeks. So the non-responsive maintainer process will maybe extend it for a total of about 10 weeks (4 weeks for non-responsive maintainer process + 6 weeks for package being orphaned).

i'm +1 to @sgallagh's suggestion, though I think we should perhaps take it case-by-case in light of @zbyszek's comment.

If we want to do it on a case-by-case basis we need someone to to the work, since it cannot be fully automated anymore.

Since this is also releng's domain, let's let @mohanboddu know about this ticket :-)

Another thing to consider for the policy is that we might not have a mass-rebuild on each release. Do we want to mandate that there should be one for every release to make sure we capture FTBFS packages?

Also whenever we retire packages, we might create packages with broken dependencies. There was a policy to clean then up in the past, too, but it was also not followed. What will we do with these packages?

Another proposal I have is instead of retiring packages is to orphan them and fix the long-time orphan packages process. Then we do not have too many processes in parallel that will retire packages, but just get broken packages be orphaned and then let the other process clean up. This would also incorporate the idea to start the non-responsive maintainer process.

About closing bugs for retiring packages: Currently there is no process to close bugs for retired packages afaik. IMHO it would be good to make this a separate process for all retired packages instead of treating the FTBFS bugs special.

Also I would like to change the process to use just one FTBFS tracker bug and file only up to one FTBFS bug per package and update it accordingly. IMHO having multiple bugs make things just more complicated/less readible. If the reason for separate trackers is to keep the amount of bugs blocking the tracker low I suggest to remove closed bugs from the tracker instead.

Another thing to consider for the policy is that we might not have a mass-rebuild on each release. Do we want to mandate that there should be one for every release to make sure we capture FTBFS packages?

I'd be against forcing a mass rebuild just for that. If there is no mass rebuild, it means gcc and glibc didn't change, so the chances of new ftbfs failures are smaller. OTOH, packages which were flagged in the previous mass rebuild should still be retired. I'll amend the proposal to include that.

Also whenever we retire packages, we might create packages with broken dependencies. There was a policy to clean then up in the past, too, but it was also not followed. What will we do with these packages?

That's why we retire early after the mass rebuild. This gives people time to work on dependent packages if the ftbfs cannot be fixed.

Another proposal I have is instead of retiring packages is to orphan them and fix the long-time orphan packages process.

Orphaning has the disadvantage that the package is still there. Dependent packages can keep using the ftbfs package. IIUC, we'd first orphan the package, and then wait a few weeks, and then retire it if no maintainer appears. This extra step would increase the time until the package is retired significantly.

Metadata Update from @zbyszek:
- Issue assigned to zbyszek

2 years ago

An updated policy proposal, taking feedback into account (changes in bold):

  1. Add a note that after each mass rebuild FTBFS bugs will be created for packages that failed.

  2. Remove all the outdated mentions of Matt Domsch and monthly rebuilds, BugZappers, CVS, PackageDB, etc.

  3. Replace "Packages with unresolved FTBFS bugs at Feature Freeze for the N+2 release will be removed" with "Packages that failed to build in two consecutive mass rebuilds will be removed 2 weeks after the second mass rebuild. If there is no mass rebuild in a given release, the time to removal is still running, and the removal will happen a week before branching.

  4. Introduce a rule that changing the FTBFS bug status to ASSIGNED/anything-else-that-is-not-NEW means that the bug is being worked on.

  5. The non-responsive maintainer policy will be started 6 weeks after a ftbfs bug is opened, unless the bug is changed as in 4. above.

I we start enforcing this policy, i.e. file bugs regularly after each mass rebuild, and actually retire packages, packages that FTBFS will go away after approximately months.

On the rel-eng side, I'd like to propose the following requirements:
- file FTBFS bugs for all packages which newly fail in each mass rebuild
- for packages which already have an open bug, update the existing bug with new information
- attach build logs to those bugs (root.log, build.log, mock.log)
- exclude any packages from retirement where the FTBFS bug is not in state NEW and there has been a change in the bug state in the last month. (This gives maintainers a way to automatically postpone the retirement.)
- actually do the retirements around 2 weeks after the second FTBFS
- if packages are retired, close the FTBFS bugs with an explanation as CLOSED/WONTFIX.

Metadata Update from @zbyszek:
- Issue tagged with: meeting

2 years ago

Introduce a rule that changing the FTBFS bug status to ASSIGNED/anything-else-that-is-not-NEW means that the bug is being worked on.

I'd like to have a protection from somebody who just flips this to ASSIGNED and then does nothing.

Introduce a rule that changing the FTBFS bug status to ASSIGNED/anything-else-that-is-not-NEW means that the bug is being worked on.

I'd like to have a protection from somebody who just flips this to ASSIGNED and then does nothing.

I would like to believe that this would be an exceptional case and that we can deal with it on an individual basis, should the need arise.

For the record, I'm +1 to this proposal as written.

An updated policy proposal, taking feedback into account (changes in bold):

Add a note that after each mass rebuild FTBFS bugs will be created for packages that failed.

Remove all the outdated mentions of Matt Domsch and monthly rebuilds, BugZappers, CVS, PackageDB, etc.

Replace "Packages with unresolved FTBFS bugs at Feature Freeze for the N+2 release will be removed" with "Packages that failed to build in two consecutive mass rebuilds will be removed 2 weeks after the second mass rebuild. If there is no mass rebuild in a given release, the time to removal is still running, and the removal will happen a week before branching.

How about: All packages that failed to build from source for six months will be retired one week before branching.

This simplifies it a lot and should have the same effect.

Introduce a rule that changing the FTBFS bug status to ASSIGNED/anything-else-that-is-not-NEW means that the bug is being worked on.

If a bug is ASSIGNED and a new mass rebuild happens, will we reset it to NEW?

The non-responsive maintainer policy will be started 6 weeks after a ftbfs bug is opened, unless the bug is changed as in 4. above.

IMHO the non-responsive maintainer procedure is wrong here, since we do not want to orphan all of the maintainers packages (the non-responsive maintainer procedure results in all packages of the maintainer being orphaned). I would change this.

If a FTBFS bug is in NEW state for 6 weeks, a weekly reminder will be added that the package will be orphaned when the bug is in NEW state for 8 weeks. if the FTBFS bug is in NEW state for 8 weeks, the package will be orphaned.

Note: orphaned packages should get retired after about six weeks

If a FTBFS bug is still open after six months four weeks before branching, it gets weekly reminders about the package being retired one week before branching.

I guess this is what you mean to happen.

I we start enforcing this policy, i.e. file bugs regularly after each mass rebuild, and actually retire packages, packages that FTBFS will go away after approximately 6½ months.

Ok, now I am lost. How do you get to 6.5 months?

IMHO we should first decide about a goal how fast we want what to happen with FTBFS packages and then figure out the details how to implement this technically. I am not sure if all the goals here align with other restrictions such as how much time is there between mass rebuilds and branching.

This was discussed in the FESCo meeting on 2018-05-18:
AGREED: The proposal from https://pagure.io/fesco/issue/1890#comment-512632 with changes in https://pagure.io/fesco/issue/1890#comment-512813 is approved. We'll hammer out any details later (+6, 0, 0)

I updated the policy in https://fedoraproject.org/wiki/Fails_to_build_from_source. I think I included all the requested corrections, but please double-check. If there is consensus, I'll also post a note to devel-announce@.

Minor suggestions for clarification:

  1. "If the bug is in your package" should be "If the build of your package fails due to a bug in your package"
  2. "If the bug is in a different package" should be "If the build of your package fails due to a bug in another package (such as a compiler bug or missing dependency)"

The way it reads right now could be confusingly interpreted as "in case a FTBFS bug is assigned to you for someone else's package". I realize that a closer read would resolve the confusion, but we can probably make that simpler on people.

I might also extend "If the package should be retired," to be "If the package is no longer useful to the Fedora project, it should be retired,".

We might add one more bullet point reminding people that if they have no time for this package, they should voluntarily orphan it immediately, rather than waiting for the automated process to do it in eight weeks (thus reducing the amount of time available to whomever takes it up).

Lastly, I think "In all cases, if you close an FTBFS bug as a duplicate of another bug, please make the other bug to block the right FTBFS tracking bugs." should be replaced by "Do not close an FTBFS bug as a duplicate, because this makes tracking difficult and may result in a new FTBFS bug being created. Instead, mark the other bug as blocking the FTBFS bug."

Thanks for the review and comments.

"If the bug is in your package" should be "If the build of your package fails due to a bug in your package"

Done.

"If the bug is in a different package" should be "If the build of your package fails due to a bug in another package (such as a compiler bug or missing dependency)"

Done.

Extend "If the package should be retired," to be "If the package is no longer useful to the Fedora project, it should be retired,"

Done.

We might add one more bullet point reminding people that if they have no time for this package, they should voluntarily orphan it immediately, rather than waiting for the automated process

Sure, but OTOH, that's always true, and I don't want to make this list longer than it has to be. If you have a succinct formulation, just add it to the page.

Lastly, I think "In all cases, if you close an FTBFS bug as a duplicate of another bug, please make the other bug to block the right FTBFS tracking bugs." should be replaced by "Do not close an FTBFS bug as a duplicate, because this makes tracking difficult and may result in a new FTBFS bug being created. Instead, mark the other bug as blocking the FTBFS bug."

I think that it's sometimes still to mark as DUPLICATE: when a FTBFS bug was already created, and a new one is made automatically. I think in that case it's better to keep using the original one, and close the new one. I put "Only one FTBFS bug should be open at any time against a given package.", and I think it's better to generally keep the amount of open bugs down to the necessary minimum.

(It's not clear how to identify all the FTBFS bugs. Does "FTBFS" have to appear in the title? Is it enough to block the tracking bug? This is important, but I'd leave it out for now and ask releng to document the proper way.)

What about this text for devel-announce@:

"""
Fedora package maintainers,

FESCo approved an updated policy for packages which fail to build from
source during mass rebuilds (FTBFS) [1].

The updated policy is still at https://fedoraproject.org/wiki/Fails_to_build_from_source.

Highlights:

  • packages which FTBFS are subject to orphaning if there is no
    maintainer acknowledgement within 8 weeks

  • packages which FTBFS in two consecutive mass rebuilds will be
    retired soon after the second mass rebuild

The implementation of this policy hinges on improving the releng
scripts used to create and manage FTBFS bugs. There is approximately
two months until the next use of those scripts, so I'm hopeful we'll
get them working.

If your package wasn't successfully built for F28, please fix that!

[1] https://pagure.io/fesco/issue/1890
[2] https://pagure.io/fesco/issue/1877#comment-509161
"""

The announcement text looks fine to me. +1

This was discussed in the FESCo meeting yesterday, and the text above was accepted.

The announcement:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/P3KFTJMDNO42POS5N3Z4UXDNPFGAQH73/

Metadata Update from @zbyszek:
- Issue untagged with: meeting
- Issue status updated to: Closed (was: Open)

2 years ago

I rerun my script to close bugs. This time, unlike previously, I also closed any bugs in state NEW or ASSIGNED where there was a successful build in F28. My assumption is that people might not want to release updates this late after a release, for example if the fix involved a version bump, and even if they should, the FTBFS is not the place to track that. I also closed, like previously, any bugs in state NEW or ASSIGNED for which an update exists, but wasn't marked in bodhi for this bug.

This brings down the list of open bugs from 764 to 643.
So the NEW/ASSIGNED list is now cleaned up and only includes packages that actually were not built.

The question is how to implement the rest of the policy.
My proposal is to do the following:

  • use the bugzilla mass bug dialogue to
    • set the NEEDINFO flag for bug assignees
    • add a comment (proposed text below).
  • after 8 weeks, the orphaning can be performed for those packages for which are still on the NEW list.

The mass rebuild kicks in next week, new FTBFS bugs will be opened.

  • after the mass rebuild is done and any immediate fallout has been fixed, this procedure can be repeated.

This way, the NEEDINFO flag will serve to deliver the weekly notices to maintainers.

proposed text:
Dear Maintainer,
your package has not been built successfully in F28. Action is required from you.
If you can fix your package to build, perform a build in koji, and either create an update
in bodhi, or close this bug without creating an update, if updating is not appropriate.
If you are working on a fix, set the status to ASSIGNED to acknowledge this.
Following the latest policy for such packages [1], your package will be orphaned
if this bug remains in NEW state more than 8 weeks.

If nobody objects, I'll do this.

Don't forget to add the [1] link to the Bugzilla comment.

Text:

Dear Maintainer,

your package has not been built successfully in F28. Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. Following the latest policy for such packages [2], your package
will be orphaned if this bug remains in NEW state more than 8 weeks.

[1] https://fedoraproject.org/wiki/Updates_Policy
[2] https://fedoraproject.org/wiki/Fails_to_build_from_source#Package_Removal_for_Long-standing_FTBFS_bugs

Bugzilla query with the bug list:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1555994%2C1555867%2C1556173%2C1582767%2C1556343%2C1556353%2C1556183%2C1556270%2C1556473%2C1556336%2C1555503%2C1556087%2C1556196%2C1583305%2C1555643%2C1555641%2C1556409%2C1555566%2C1556419%2C1555907%2C1556231%2C1556517%2C1556390%2C1556302%2C1555852%2C1555529%2C1556314%2C1555797%2C1556219%2C1556077%2C1583358%2C1555730%2C1556415%2C1582922%2C1556496%2C1555546%2C1556367%2C1555848%2C1556134%2C1582781%2C1555803%2C1415788%2C1556566%2C1555777%2C1583365%2C1556059%2C1556465%2C1556625%2C1555605%2C1583063%2C1555490%2C1556349%2C1556399%2C1556104%2C1555978%2C1555953%2C1582766%2C1556407%2C1556128%2C1556052%2C1582753%2C1556374%2C1555541%2C1556321%2C1556355%2C1555587%2C1555826%2C1555833%2C1555814%2C1556452%2C1555800%2C1556384%2C1555960%2C1556514%2C1555967%2C1556498%2C1555985%2C1582953%2C1556251%2C1556539%2C1556111%2C1555780%2C1556489%2C1555514%2C1555855%2C1555779%2C1556335%2C1555652%2C1555817%2C1556472%2C1555875%2C1556373%2C1556137%2C1583507%2C1556010%2C1556157%2C1555642%2C1582768%2C1556308%2C1583336%2C1556414%2C1555776%2C1556076%2C1556494%2C1582755%2C1555821%2C1555493%2C1556095%2C1555822%2C1583392%2C1555868%2C1583372%2C1583294%2C1556523%2C1556294%2C1555948%2C1582776%2C1556599%2C1555606%2C1556023%2C1556538%2C1556103%2C1556178%2C1555884%2C1583124%2C1556365%2C1555589%2C1582771%2C1556641%2C1555832%2C1555902%2C1556617%2C1555613%2C1556418%2C1555553%2C1555966%2C1556149%2C1556502%2C1555790%2C1583273%2C1556385%2C1555491%2C1555768%2C1583368%2C1556422%2C1583061%2C1555581%2C1556254%2C1555576%2C1556250%2C1555588%2C1555543%2C1555585%2C1583373%2C1583296%2C1555708%2C1556573%2C1556561%2C1582773%2C1555480%2C1583127%2C1556253%2C1556400%2C1556217%2C1583083%2C1583311%2C1556284%2C1556109%2C1583057%2C1556136%2C1556061%2C1555679%2C1555972%2C1556042%2C1556174%2C1556354%2C1556392%2C1555484%2C1555653%2C1556096%2C1555570%2C1583291%2C1555808%2C1583411%2C1555860%2C1556352%2C1555551%2C1583384%2C1555709%2C1556364%2C1555866%2C1556516%2C1582928%2C1556323%2C1556218%2C1555752%2C1556454%2C1555887%2C1556474%2C1555901%2C1556509%2C1556337%2C1555677%2C1582908%2C1556375%2C1583366%2C1556181%2C1556345%2C1556032%2C1556188%2C1555649%2C1555851%2C1555854%2C1556199%2C1555772%2C1555876%2C1555946%2C1555990%2C1556151%2C1555809%2C1556113%2C1555665%2C1424182%2C1555956%2C1556410%2C1556148%2C1582778%2C1555685%2C1556234%2C1556123%2C1556446%2C1555707%2C1555656%2C1556342%2C1583310%2C1556268%2C1556542%2C1556416%2C1556583%2C1555766%2C1555828%2C1556574%2C1556397%2C1556101%2C1556293%2C1555991%2C1556084%2C1583118%2C1556444%2C1583383%2C1555670%2C1556167%2C1556484%2C1555751%2C1556357%2C1556179%2C1556430%2C1556074%2C1556274%2C1555909%2C1555483%2C1556041%2C1556437%2C1556387%2C1555968%2C1556497%2C1556299%2C1555841%2C1555815%2C1556326%2C1556591%2C1556445%2C1555798%2C1555691%2C1556197%2C1556158%2C1582777%2C1556528%2C1582909%2C1556256%2C1556579%2C1556619%2C1556549%2C1556475%2C1555971%2C1555725%2C1582914%2C1556481%2C1555874%2C1556130%2C1555969%2C1556025%2C1592492%2C1555535%2C1583035%2C1556056%2C1556411%2C1555820%2C1555888%2C1556053%2C1555955%2C1556003%2C1556447%2C1556124%2C1555518%2C1555572%2C1556359%2C1555877%2C1555819%2C1556402%2C1556536%2C1556490%2C1555601%2C1556090%2C1556507%2C1555881%2C1583039%2C1555477%2C1556439%2C1555650%2C1555524%2C1555747%2C1555806%2C1556246%2C1556017%2C1555810%2C1555507%2C1556100%2C1556513%2C1556310%2C1556476%2C1555889%2C1556094%2C1555750%2C1556063%2C1556230%2C1583066%2C1556356%2C1556039%2C1556083%2C1556086%2C1556462%2C1555618%2C1556147%2C1556263%2C1583286%2C1555651%2C1556324%2C1583390%2C1556456%2C1583361%2C1555630%2C1556055%2C1583040%2C1555959%2C1556089%2C1583279%2C1555563%2C1555825%2C1555757%2C1555981%2C1555904%2C1556316%2C1583409%2C1555494%2C1583406%2C1555958%2C1556545%2C1556370%2C1555740%2C1556070%2C1555829%2C1555534%2C1556333%2C1556176%2C1556060%2C1555962%2C1583386%2C1556377%2C1555950%2C1556160%2C1555748%2C1583284%2C1583387%2C1555578%2C1556028%2C1555562%2C1556079%2C1555906%2C1555489%2C1556491%2C1555676%2C1556309%2C1555699%2C1555476%2C1556401%2C1556506%2C1583120%2C1556262%2C1556064%2C1583270%2C1556581%2C1556012%2C1556058%2C1556088%2C1555573%2C1556038%2C1555792%2C1556044%2C1555844%2C1556511%2C1556611%2C1555756%2C1582902%2C1556242%2C1556040%2C1556289%2C1556396%2C1556358%2C1555952%2C1555979%2C1582748%2C1556227%2C1583278%2C1583125%2C1556386%2C1556376%2C1555502%2C1556027%2C1556431%2C1556369%2C1556469%2C1555741%2C1555622%2C1556508%2C1555702%2C1555949%2C1556580%2C1556477%2C1556621%2C1583357%2C1556080%2C1555903%2C1556244%2C1556237%2C1555559%2C1555957%2C1583309%2C1556115%2C1556404%2C1556344%2C1555961%2C1583364%2C1556565%2C1556413%2C1556361%2C1555658%2C1555989%2C1555731%2C1555758%2C1555850%2C1555783%2C1556623%2C1556307%2C1556505%2C1555479%2C1556057%2C1556471%2C1583292%2C1556595%2C1555795%2C1556304%2C1556065%2C1556317%2C1556194%2C1556546%2C1556005%2C1555892%2C1555488%2C1555788%2C1556346%2C1555869%2C1583295%2C1555623%2C1556334%2C1556492%2C1555659%2C1556280%2C1583356%2C1555668%2C1556185%2C1556372%2C1556213%2C1555705%2C1555802%2C1555975%2C1556488%2C1582780%2C1555879%2C1556540%2C1556331%2C1555742%2C1555567%2C1556495%2C1556037%2C1555885%2C1556098%2C1556214%2C1556457%2C1583367%2C1555835%2C1555853%2C1556312%2C1556451%2C1556019%2C1556338%2C1583378%2C1556553%2C1556389%2C1556127%2C1556330%2C1556066%2C1556421%2C1556339%2C1555645%2C1556318%2C1582933%2C1555724%2C1555965%2C1555595%2C1556368%2C1555836%2C1556463%2C1556341%2C1556116%2C1556468%2C1556297%2C1582932%2C1582763%2C1583381%2C1556403%2C1555538%2C1556351%2C1583060%2C1555951%2C1555805%2C1556203%2C1556442%2C1555763%2C1555700%2C1424037%2C1555890%2C1555666%2C1556220%2C1555687%2C1555762%2C1556119%2C1556332%2C1582893%2C1556276%2C1555754%2C1556030%2C1555988%2C1583401%2C1556504%2C1555970%2C1555743%2C1555759%2C1583267%2C1556535%2C1555842%2C1555774%2C1556503%2C1555905%2C1556441%2C1555789%2C1556298%2C1518800%2C1556512%2C1556046%2C1555908%2C1556135%2C1555723%2C1556301%2C1556029%2C1556371%2C1556633%2C1555873%2C1583359%2C1555893%2C1555811%2C1555787%2C1583377%2C1556620%2C1556106%2C1556379%2C1583380%2C1556117%2C1583362%2C1583275%2C1555982%2C1556340%2C1555673%2C1556388%2C1555900%2C1556239%2C1556450%2C1555542%2C1556078%2C1582934%2C1555954%2C1556458%2C1556398%2C1556493%2C1556420%2C1583069%2C1583370%2C1555974%2C1555963%2C1582769%2C1555824%2C1555807%2C1555781%2C1555862%2C1556099%2C1555794%2C1556071%2C1582918%2C1556350%2C1556329%2C1555683%2C1556406%2C1555667%2C1556440&bug_id_type=anyexact&bug_status=NEW&list_id=9089812&query_format=advanced

Sample comment:
https://bugzilla.redhat.com/show_bug.cgi?id=1583392#c4

Pfff, I screwed up. The bugzilla query to update the bugs timed out, and I thought I can use the severity field (which I changed to "high" in the query), to filter only those bugs which didn't get updated. But it turns out they were being updated in the background or something. In the end some bugs have the needinfo flag and comment more than one time. If you are affected by this, sorry for the noise.

Login to comment on this ticket.

Metadata