#7718 The FTBFS cleanup policy is not happening
Opened a year ago by churchyard. Modified 17 days ago

The https://fedoraproject.org/wiki/Fails_to_build_from_source policy says:

If a FTBFS bug remains in NEW state, a weekly reminder will be sent.

This never happened.

(Opened also as https://pagure.io/fesco/issue/1973 - feel free to close the less appropriate one.)


Metadata Update from @mohanboddu:
- Issue tagged with: meeting

a year ago

Metadata Update from @mohanboddu:
- Issue assigned to mohanboddu

a year ago

We dont have the tooling ready for it, I will work on it asap.

@zbyszek is also working on this and currently uses the Bugzilla NEEDINFO reminders for this.

The package will be orphaned after the FTBFS bug is in NEW state for 8 weeks.

This should happen this week. No remainders have been sent so far.

Is the orphaning still happening or is it blocked by the reminders?

I suppose the orphaning is blocked and it also seems that the F29 FTBFS bugs are not even set to needinfo: https://bugzilla.redhat.com/showdependencytree.cgi?id=1602938&hide_resolved=1

@zbyszek can you maybe post an update?

I am sorry that I poke at this again and again, but is there anything I can do to move this forward?

I can send the reminders myself, if this would make the policy happen. Let me know.

There are 1042 FTBFS bugzillas in NEW state. Those packages should have been orphaned already by this policy, yet nothing happened yet.

Getting unmaintained packages orphaned (and later retired) is important to get rid of old cruft, for me personally this is relevant for the "get rid of python2" effort, where plenty of packages that use python2 (yet nobody updates them) would go away. I also believe this is needed for the general health of the project.

This appears to be the consensus of FESCo anyway https://pagure.io/fesco/issue/1890

@churchyard Can you create a script to do this and send it as PR. As I dont have time for this, I would appreciate your help.

Thanks

I'll have a look in couple days.

Just to make sure, the script will:

Here's a script:
https://github.com/hroncok/ftbfs_bugzilla/blob/master/ftbfs_weekly_reminder.py - it is public domain.

It updates bugz one by one and maintains a list of already updated bugz (at filesystem level) = it can be interrupted and run again.

I guess updating in bulk would be faster, but:

  • I found no way to set needinfo for assignee in bulk
  • it tends to error and there is no (easy) way to tell which bugs got updated

So this is slow but quite reliable.

There is also bugzilla.log that gives you all the debug information needed.

The script has a TESTING boolean, set to True if you really want to run this, keep at False to use partner-bugzilla and fake bugz. Uses click and python-bugzilla (available from PyPI or RPMs). Tested on Python 3.7 only, should work with 3.6 as well.

Let me know if this is not what you wanted.

Is the script OK for what you needed? How can I help? Shall I run it myself?

A week before the mass branching any packages which still have open FTBFS bugs from the previous release will be retired.

That shall be 2019-02-12.

We have 1043 open F29 FTBFS bugs. Are we going to retire 1043 packages in February?

We have a new FTBFS tracker for a new release. Can we get this started please? How can I help?

So, we need to set this up in infrastructure.

It looks like it uses an interactive login? Can we have it take a user/token for running in cron?

I guess we also need to determine what user we want to use here... releng? bodhi? something new for just this use?

It looks like it uses an interactive login? Can we have it take a user/token for running in cron?

Do you have an example of such script so I don't need to reinvent the wheel?

I guess we also need to determine what user we want to use here...

Releng files those bugs, so I guess we can reuse that user as well.

from turbogears import config, yummy :(

Do you have an example config I can see to rewrite this without turbogears? I guess I can use plain ConfigParser.

I'm going to assume this for now:

[bugzilla]
username = releng
password = hunter2

Adjusted https://pagure.io/releng/pull-request/7881 to use a config file. Adjust the config file paths as desired.

Can we get the cron set up please? I'll try to talk to @kevin on IRC on Monday, so we can work on it together.

Kevin is out on vacation until the 17th. I will try and look at what needs to be done on Monday.

Mohan and I deployed this this morning. It should start bugging people on the weekly cron.weekly run

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 months ago

While I have seen some of the comments, last time they have happened at 2019-06-17. This is apparently not happening weekly.

Take for example this randomly selected F30FTBFS bug:

The comments happened:

  • 2019-04-27 (possibly a test run)
  • 2019-06-03
  • 2019-06-17 (2 weeks after the previous, not 1 week)

If more comments appear today, we probably just run it bi-weekly instead of weekly.

Metadata Update from @churchyard:
- Issue status updated to: Open (was: Closed)

2 months ago

Cron job script was hosed. I have fixed the script and I am now running the script by hand.

OK running the script, it gave the following output:

[root@compose-x86-01 ~][PROD]# su ftbfs /etc/cron.weekly/ftbfs.cron 
Cloning into 'releng'...
remote: Counting objects: 7869, done.
remote: Compressing objects: 100% (3487/3487), done.
remote: Total 7869 (delta 4422), reused 7712 (delta 4286)
Receiving objects: 100% (7869/7869), 1.36 MiB | 1.89 MiB/s, done.
Resolving deltas: 100% (4422/4422), done.
Loading bug IDs from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED. Will not fill those. Remove /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED to stop this from happening.
Gathering bugz, this can take a while...
There are 755 NEW bugz, will send a reminder
Will ignore 758 bugz from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED
Will update 25 bugz
  [####################################]  100%  Finished!                                                    
Updated 0 bugz
Failed to update 25 bugz
https://bugzilla.redhat.com/show_bug.cgi?id=1674669
https://bugzilla.redhat.com/show_bug.cgi?id=1674680
https://bugzilla.redhat.com/show_bug.cgi?id=1674753
https://bugzilla.redhat.com/show_bug.cgi?id=1674783
https://bugzilla.redhat.com/show_bug.cgi?id=1674817
https://bugzilla.redhat.com/show_bug.cgi?id=1675022
https://bugzilla.redhat.com/show_bug.cgi?id=1675036
https://bugzilla.redhat.com/show_bug.cgi?id=1675097
https://bugzilla.redhat.com/show_bug.cgi?id=1675108
https://bugzilla.redhat.com/show_bug.cgi?id=1675364
https://bugzilla.redhat.com/show_bug.cgi?id=1675407
https://bugzilla.redhat.com/show_bug.cgi?id=1675415
https://bugzilla.redhat.com/show_bug.cgi?id=1675454
https://bugzilla.redhat.com/show_bug.cgi?id=1675586
https://bugzilla.redhat.com/show_bug.cgi?id=1675593
https://bugzilla.redhat.com/show_bug.cgi?id=1675684
https://bugzilla.redhat.com/show_bug.cgi?id=1675784
https://bugzilla.redhat.com/show_bug.cgi?id=1675892
https://bugzilla.redhat.com/show_bug.cgi?id=1675899
https://bugzilla.redhat.com/show_bug.cgi?id=1675918
https://bugzilla.redhat.com/show_bug.cgi?id=1676017
https://bugzilla.redhat.com/show_bug.cgi?id=1676164
https://bugzilla.redhat.com/show_bug.cgi?id=1676181
https://bugzilla.redhat.com/show_bug.cgi?id=1676205
https://bugzilla.redhat.com/show_bug.cgi?id=1679955

I am not sure what is causing bzapi.update_bugs([bug.id], update) to give an exception.

rerunning the script on failure should do.

It seemed to fail identically on rerun

[root@compose-x86-01 ~][PROD]# su ftbfs /etc/cron.weekly/ftbfs.cron 
Cloning into 'releng'...
remote: Counting objects: 7869, done.
remote: Compressing objects: 100% (3487/3487), done.
remote: Total 7869 (delta 4422), reused 7713 (delta 4286)
Receiving objects: 100% (7869/7869), 1.36 MiB | 1.63 MiB/s, done.
Resolving deltas: 100% (4422/4422), done.
Loading bug IDs from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED. Will not fill those. Remove /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED to stop this from happening.
Gathering bugz, this can take a while...
There are 755 NEW bugz, will send a reminder
Will ignore 758 bugz from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED
Will update 25 bugz
  [####################################]  100%  Finished!                                                    
Updated 0 bugz
Failed to update 25 bugz
https://bugzilla.redhat.com/show_bug.cgi?id=1674669
https://bugzilla.redhat.com/show_bug.cgi?id=1674680
https://bugzilla.redhat.com/show_bug.cgi?id=1674753
https://bugzilla.redhat.com/show_bug.cgi?id=1674783
https://bugzilla.redhat.com/show_bug.cgi?id=1674817
https://bugzilla.redhat.com/show_bug.cgi?id=1675022
https://bugzilla.redhat.com/show_bug.cgi?id=1675036
https://bugzilla.redhat.com/show_bug.cgi?id=1675097
https://bugzilla.redhat.com/show_bug.cgi?id=1675108
https://bugzilla.redhat.com/show_bug.cgi?id=1675364
https://bugzilla.redhat.com/show_bug.cgi?id=1675407
https://bugzilla.redhat.com/show_bug.cgi?id=1675415
https://bugzilla.redhat.com/show_bug.cgi?id=1675454
https://bugzilla.redhat.com/show_bug.cgi?id=1675586
https://bugzilla.redhat.com/show_bug.cgi?id=1675593
https://bugzilla.redhat.com/show_bug.cgi?id=1675684
https://bugzilla.redhat.com/show_bug.cgi?id=1675784
https://bugzilla.redhat.com/show_bug.cgi?id=1675892
https://bugzilla.redhat.com/show_bug.cgi?id=1675899
https://bugzilla.redhat.com/show_bug.cgi?id=1675918
https://bugzilla.redhat.com/show_bug.cgi?id=1676017
https://bugzilla.redhat.com/show_bug.cgi?id=1676164
https://bugzilla.redhat.com/show_bug.cgi?id=1676181
https://bugzilla.redhat.com/show_bug.cgi?id=1676205
https://bugzilla.redhat.com/show_bug.cgi?id=1679955
ftbfs_weekly_reminder.py had an error condition
Look in /tmp/ftbfs_reminder.8AGrhM for more info

Here is the script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/bin/bash                                                                                                                                                                                   
# A weekly cron job which actually does the items needed.                                                                                                                                     
TMPDIR=`mktemp -d /tmp/ftbfs_reminder.XXXXXX`
GITREPO=https://pagure.io/releng.git
SCRIPT=ftbfs_weekly_reminder.py
if [ $? -eq 0 ]; then
    cd ${TMPDIR}
    git clone ${GITREPO}
    if [ $? -eq 0 ]; then
        cd releng/scripts
        ./${SCRIPT}
        if [ $? -ne 0 ]; then
            echo "${SCRIPT} had an error condition"
            echo "Look in ${TMPDIR} for more info"
            # Do not clean up trash                                                                                                                                                           
            exit 1
        fi
    else
        echo "Unable to clone ${GITREPO}"
        echo "Look in ${TMPDIR} for more info"
        # Do not clean up trash                                                                                                                                                               
        exit 1
    fi
    cd /tmp/
    rm -rf $TMPDIR
else
    echo "Unable to create ${TMPDIR}"
    exit 1
fi

You can't ask Marek Mahut mmahut@redhat.com because that account is disabled.

It seems that the script needs to distinguish temporary BZ nonsense failures against permanent ones :(

do you happen to have /tmp/ftbfs_reminder.8AGrhM ? the script loggs the exceptions.

correction, the exception is logged in bugzilla.log

Traceback (most recent call last):
  File "ftbfs_weekly_reminder.py", line 90, in send_reminder
    bzapi.update_bugs([bug.id], update)
  File "/usr/lib/python3.7/site-packages/bugzilla/base.py", line 1321, in update_bugs
    return self._proxy.Bug.update(tmp)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 104, in _ServerProxy__request
    self)._ServerProxy__request(methodname, params)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 201, in request
    return self._request_helper(url, request_body)
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 179, in _request_helper
    return self.parse_response(response)
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 154, in parse_response
    return unmarshaller.close()
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 32000: "You can't ask Martin Preisler <mpreisle@redhat.com> because that account is disabled.">

See https://pagure.io/releng/pull-request/8480 - it now attempts not to add needinfo if it breaks. The script output is also useful in case we want to see what redhat bugzilla accounts are gone.

Metadata Update from @churchyard:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

https://pagure.io/releng/issue/8522 is the request to retire FTBFS packages according to the policy

I have a feeling that this stopped happening again.

Metadata Update from @churchyard:
- Issue status updated to: Open (was: Closed)

2 months ago

To clarify, the comments happened on 2019-07-02 and than stopped. No comments a week later, no comments 2 weeks later either.

I made a spalling mistake in the cron job so it was not able to run. I have corrected the cron job and run the script by hand.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Metadata Update from @smooge:
- Issue status updated to: Open (was: Closed)

2 months ago

Script failed to run:

[root@compose-x86-01 ~][PROD]# su -s /bin/sh ftbfs
sh-5.0$ sh -x /usr/local/bin/ftbfs.sh 
++ mktemp -d /tmp/ftbfs_reminder.XXXXXX
+ TMPDIR=/tmp/ftbfs_reminder.jutVAp
+ GITREPO=https://pagure.io/releng.git
+ SCRIPT=ftbfs_weekly_reminder.py
+ '[' 0 -eq 0 ']'
+ cd /tmp/ftbfs_reminder.jutVAp
+ git clone https://pagure.io/releng.git
Cloning into 'releng'...
remote: Counting objects: 7902, done.
remote: Compressing objects: 100% (3401/3401), done.
remote: Total 7902 (delta 4449), reused 7835 (delta 4405)
Receiving objects: 100% (7902/7902), 1.36 MiB | 741.00 KiB/s, done.
Resolving deltas: 100% (4449/4449), done.
+ '[' 0 -eq 0 ']'
+ cd releng/scripts
+ ./ftbfs_weekly_reminder.py
Gathering bugz, this can take a while...
There are 681 NEW bugz, will send a reminder
Traceback (most recent call last):
  File "./ftbfs_weekly_reminder.py", line 126, in <module>
    if ignore:
NameError: name 'ignore' is not defined
+ '[' 1 -ne 0 ']'
+ echo 'ftbfs_weekly_reminder.py had an error condition'
ftbfs_weekly_reminder.py had an error condition
+ echo 'Look in /tmp/ftbfs_reminder.jutVAp for more info'
Look in /tmp/ftbfs_reminder.jutVAp for more info
+ exit 1

Metadata Update from @kevin:
- Issue tagged with: backlog

17 days ago

Login to comment on this ticket.

Metadata