Issue #5473: Upcoming 5.14 release - pagure

To-Do:
- Final test of release branch
- Pushing security fixes to master and update 5.14.1 changelog
- Pushing final state of release branch to PR and merge
- Tag last commit in 5.14.x branch with 5.14.1 and publish release
- Submit update for Fedora pagure package
- Announce new intermediate release

Edited 21 days ago by wombelix

kevin commented 22 days ago

From the fedora infra view, epel8 is the important thing, and rhel 8.10 just dropped today... might be a good idea to make sure none of the 8.10 changes broke anything. :)

wombelix commented 22 days ago

From the fedora infra view, epel8 is the important thing, and rhel 8.10 just dropped today... might be a good idea to make sure none of the 8.10 changes broke anything. :)

Thanks for your Feedback @kevin, appreciated.

I wonder how pagure runs in fedora infra, via the rpm package (https://src.fedoraproject.org/rpms/pagure) or as a pip/venv installation? Are they any other customizations done or workarounds or patches applied? Because the rpm package only contains a couple backported features that will be part of 5.14.1 but nothing specific to EL8 or any other OS.

Even though I branched from 5.13.x and backported quite a lot of fixes, don't expect too much of this - rather unplanned - intermediate 5.14.1 release. It's supposed to bridge the gap till release 6.0 a bit. It still contains python2 code, outdated dependencies, fails unit-tests and all those know tech-dept challenges. The included fixes address general bugs or security problems, but are not tailored to a specific OS or Python version. So at the end, 5.14.1 will work as good and bad on a system that currently runs 5.13.3.

I did run the unit tests against the 5.14.1 release branch on EL8 today after applying one monkey patch to fix a flask-wtf bug: 284 failed, 1440 passed, 6 skipped, 6 warnings
Some are cosmetic (black/flake8 formatting findings). Others are related to the gitolite backend which shouldn't be in use anymore. A majority is ContextualVersionConflict because some components seem to have dependencies to newer version of modules that are not available on EPEL 8.

All this will be addressed with the 6.0 release that I can start to prepare as soon 5.14.1 is out. But getting the dependencies in order and all unit tests fixed on a broad range of python versions and module versions is something we had to give up. On the master branch and therefore for all release from 6.0 onwards, the oldest python version we run currently tests against is 3.9 for example. Which means from an EL perspective, 9 and above.

kevin commented 21 days ago

From the fedora infra view, epel8 is the important thing, and rhel 8.10 just dropped today... might be a good idea to make sure none of the 8.10 changes broke anything. :)

Thanks for your Feedback @kevin, appreciated.

I wonder how pagure runs in fedora infra, via the rpm package (https://src.fedoraproject.org/rpms/pagure) or as a pip/venv installation? Are they any other customizations done or workarounds or patches applied? Because the rpm package only contains a couple backported features that will be part of 5.14.1 but nothing specific to EL8 or any other OS.

we have 4 installs (2 prod, 2 stg, 2 pagure and 2 pagure+pagure-dist-git). All of them are running the epel8 version:

We do have one hotfix:

# httplib2 0.10.3, as available in EL8, hardcodes TLSv1. The next version
# (0.11.0) chooses the TLS version that is supported by both client and server.
# That variable is only available since Python 3.5+, but EL8 has Python 3.6, so
# we're good to patch. The alternative would be for EL8 to update httplib2 to
# at least version 0.11.0.
- name: Patch httplib2 to not hardcode TLSv1, which is not accepted by Ipsilon
  ansible.builtin.replace:
    path: /usr/lib/python3.6/site-packages/httplib2/__init__.py
    regexp: 'ssl\.PROTOCOL_TLSv1'
    replace: 'ssl.PROTOCOL_TLS'

So happy to test a proposed epel8 rpm in staging...

Understood on the rest.

wombelix commented 20 days ago

pkgs01.iad2.fedoraproject.org | CHANGED | rc=0 | (stdout) pagure-5.13.3-10.el8.noarch
pkgs01.stg.iad2.fedoraproject.org | CHANGED | rc=0 | (stdout) pagure-5.13.3-10.el8.noarch
pagure-stg01.fedoraproject.org | CHANGED | rc=0 | (stdout) pagure-5.13.3-10.el8.noarch
pagure02.fedoraproject.org | CHANGED | rc=0 | (stdout) pagure-5.13.3-10.el8.noarch

We do have one hotfix:

\# httplib2 0.10.3, as available in EL8, hardcodes TLSv1. The next version \# (0.11.0) chooses the TLS version that is supported by both client and server. \# That variable is only available since Python 3.5+, but EL8 has Python 3.6, so \# we're good to patch. The alternative would be for EL8 to update httplib2 to \# at least version 0.11.0. - name: Patch httplib2 to not hardcode TLSv1, which is not accepted by Ipsilon ansible.builtin.replace: path: /usr/lib/python3.6/site-packages/httplib2/__init__.py regexp: 'ssl\.PROTOCOL_TLSv1' replace: 'ssl.PROTOCOL_TLS'

So happy to test a proposed epel8 rpm in staging...

OK, I published the release 5.14.1 an hour ago and updated the Fedora package. This is the EPEL8 build: https://koji.fedoraproject.org/koji/buildinfo?buildID=2456426

The hotfix you have is independent of the pagure source and package so you can just keep it in the ansible playbook :)

Edited 20 days ago by wombelix

wombelix commented 20 days ago

All tasks complete, 5.14.1 release done.
Following details for documentation purposes.

[x] Final test of release branch:
Package builds on rawhide and epel8. I applied one hotfix to the unit tests to keep EL8 a bit more happy. Based on the backported changes I'm confident 5.14.1 will work as good and bad as 5.13.3 on systems.

[x] Pushing security fixes to master and update 5.14.1 changelog
- fix_security_issue_rhbz2277121_argument_injection: https://pagure.io/pagure/pull-request/5481
- fix_security_issue_rhbz2280030_generate_archive_follows_symlink: https://pagure.io/pagure/pull-request/5482
- fix_security_issue_rhbz2280725_rhbz2278745_follow_symlink: https://pagure.io/pagure/pull-request/5483
- fix_security_issue_rhbz2280728_rhbz2279411_path_traversal: https://pagure.io/pagure/pull-request/5484

[x] Pushing final state of release branch to PR and merge

version bumped in: pagure/__init__.py
version bumped in: files/pagure.spec
changelog updated with security fix entries

[x] Tag last commit in 5.14.x branch with 5.14.1 and publish release

git tag -a 5.14.1 6b06ac5 -m "Release 5.14.1"
git push upstream 5.14.1

[x] Submit update for Fedora pagure package
https://src.fedoraproject.org/rpms/pagure/pull-request/8

[x] Announce new intermediate release
EMail: pagure-announce@lists.pagure.io; pagure-devel@lists.pagure.io; devel@lists.fedoraproject.org;
Matrix: #pagure:fedora.im

Announcement: pagure release 5.14.1 available

Hi everyone,

it's been a while and because of increasing requests and important security fixes we release pagure version 5.14.1 today!

The release is build based on version 5.13.3 with 59 backports: 
- 13 features, 34 fixes, 4 security fixes and 8 improvements of the documentation.

Checkout the changelog [1] for a full list with a brief description and links to the related pull request.

I want to thank everyone who contributed [2] to pagure and made this release possible!

You can download the source [3] and soon the update will arrive in the Fedora package [4] too.

Please keep in mind that 5.14.1 doesn't contain breaking changes and is sort of an intermediate release.
This means that the code base still contains python 2 code for example.
That we still have some tech-dept and failing unit tests.
Also it will most likely not run without issues on latest operating system and python versions.
But where ever you run 5.13.3 today, then 5.14.1 will run on that system too :)

All this will be addressed with the upcoming major release 6.0.
This release will need a bit more time, so stay tuned for updates on that!

Dom


[1] https://pagure.io/pagure/blob/6b06ac585529c3087364a5ebe4fb9d7c20e3c872/f/doc/changelog.rst
[2] https://pagure.io/pagure/blob/6b06ac585529c3087364a5ebe4fb9d7c20e3c872/f/doc/contributors.rst
[3] https://pagure.io/pagure/releases
[4] https://src.fedoraproject.org/rpms/pagure

[x] Update pagure Fedora package
- PR: https://src.fedoraproject.org/rpms/pagure/pull-request/8
- Rawhide build: https://koji.fedoraproject.org/koji/buildinfo?buildID=2456422
- Rawhide update: https://bodhi.fedoraproject.org/updates/FEDORA-2024-a1a85a03fa

fedpkg switch-branch f39 && git merge rawhide && git push && fedpkg build --nowait

Building pagure-5.14.1-1.fc39 for f39-candidate
Created task: 118073867
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=118073867

fedpkg switch-branch f40 && git merge rawhide && git push && fedpkg build --nowait

Building pagure-5.14.1-1.fc40 for f40-candidate
Created task: 118073870
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=118073870

fedpkg switch-branch epel8 && git merge rawhide && git push && fedpkg build --nowait

Building pagure-5.14.1-1.el8 for epel8-candidate
Created task: 118074013
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=118074013

fedpkg switch-branch f39 && fedpkg update

https://bodhi.fedoraproject.org/updates/FEDORA-2024-c77b41f037

fedpkg switch-branch f40 && fedpkg update

https://bodhi.fedoraproject.org/updates/FEDORA-2024-70b911b92d

fedpkg switch-branch epel8 && fedpkg update

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-284ac5147f

pingou commented 18 days ago

[x] Tag last commit in 5.14.x branch with 5.14.1 and publish release
git tag -a 5.14.1 6b06ac5 -m "Release 5.14.1" git push upstream 5.14.1

FYI, I used/use to gpg sign the tags.

Also, I have a script that I used to make release:

~/bin/make-release.py -v 5.13.3

Here is the script:

#!/usr/bin/env python3

"""
Adjust all files to make a release.

"""

import argparse
import datetime
import os
import re
import subprocess as sp
import sys

import arrow
import requests

from distutils.version import LooseVersion


AUTHOR = 'Pierre-Yves Chibon <pingou@pingoured.fr>'


class MakeReleaseException(Exception):
    ''' Generic Exception class for this script. '''
    pass


def run(cmd, stdout=sp.PIPE, stderr=sp.PIPE):

    print(cmd)
    proc = sp.Popen([cmd], shell=True, stdout=stdout, stderr=stderr)
    output = proc.communicate()[0]
    if stdout and stderr:
        return output.decode('utf-8').strip().split('\n')


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('-v', '--version', default=None,
                        help='New version being released')

    args = parser.parse_args()

    if not args.version:
        sys.stderr.write('A version must be provided.\n')
        sys.exit(1)

    return args


def update_version(version):
    ''' Find the file where __version__ is set and update it. '''
    ignore_files = [
       "pagure/hooks/files/git_multimail_upstream.py"
    ]
    cmd = 'git grep "__version__ = \'"'
    files = run(cmd)
    cmd = "git grep '__version__ = \"'"
    files.extend(run(cmd))
    left = []
    for filename in files:
        if not filename.strip():
            continue
        if filename.startswith('doc'):
            continue
        print(filename)
        print(filename.split(':', 1))
        if not filename.split(':', 1)[1].startswith('__version__'):
            continue
        left.append(filename)

    for filename in ignore_files:
        for file_left in left:
            if filename in file_left:
                left.pop(left.index(file_left))

    print(left)
    if len(left) != 1:
        raise MakeReleaseException(
            'More than one file found containing the __version__:\n%s' %
            left)
    filename = left[0].split(':')[0]
    if not os.path.exists(filename):
        print('Could not find the file %s' % filename)
        return

    print('Update __version__ in %s to %s' % (filename, version))

    # Read the file in
    VERSION = re.compile(r"__version__ = '(.*?)'", re.S)
    VERSION2 = re.compile(r'__version__ = "(.*?)"', re.S)
    output = []
    with open(filename) as stream:
        for row in stream:
            if VERSION.match(row):
                row = "__version__ = '%s'\n" % version
            elif VERSION2.match(row):
                row = '__version__ = "%s"\n' % version
            output.append(row)
    # Update the file
    with open(filename, 'w') as stream:
        stream.write(''.join(output))


def update_contributors(version):
    ''' Update the contributors list in the documentation. '''
    cmd = 'git grep "Number of commits"'
    files = run(cmd)
    left = []
    for filename in files:
        if not filename.startswith('doc'):
            continue
        left.append(filename)
    if len(left) != 1:
        raise MakeReleaseException(
            'More than one file found listing the contributors:\n%s' % left)
    filename = left[0].split(':')[0]
    print('Updating contributors in %s' % (filename))

    cmd = 'git shortlog -s -n -e'
    tmp = run(cmd)
    contributors = []
    for contributor in tmp:
        commits, username = contributor.strip().split('\t', 1)
        temp = '%s %s %s\n' % (commits.rjust(6), ' ' * 12, username)
        contributors.append(temp)

    # Read the file in
    output = []
    cnt = 0
    with open(filename) as stream:
        for row in stream:
            if row.startswith('================='):
                cnt += 1
            if cnt == 3:
                if row.startswith('================='):
                    output.append(row)
                    output.extend(contributors)

            else:
                output.append(row)
    # Update the file
    with open(filename, 'w') as stream:
        stream.write(''.join(output))


def update_spec(version):
    ''' Update the Version in the spec and let the user update the
    changelog.
    '''
    cmd = 'git grep "Summary:"'
    files = run(cmd)
    left = set()
    for filename in files:
        if not '.spec' in filename:
            continue
        filename = filename.split(':')[0]
        left.add(filename)
    if len(left) != 1:
        raise MakeReleaseException(
            'More than one spec file found:\n%s' % left)
    filename = left.pop()
    print('Updating spec file at %s' % (filename))


    # Read the file in
    output = []
    cnt = 0
    with open(filename) as stream:
        for row in stream:
            if 'Version:' in row:
                row = '%s %s\n' % (row.rsplit(' ', 1)[0], version)
            if 'Release:' in row:
                row = '%s 1%%{?dist}\n' % (row.rsplit(' ', 1)[0])
            if '%changelog' in row:
                output.append(row)
                dt = datetime.datetime.now().strftime("%a %b %d %Y")
                row = '* %s %s - %s-1\n- Update to %s\n\n'  % (
                    dt, AUTHOR, version, version)
            output.append(row)
    # Update the file
    with open(filename, 'w') as stream:
        stream.write(''.join(output))

    run('vim %s' % filename, stdout=None)


def main(version):
    # Step 1: update the __version__
    update_version(version)
    try:
        # Step 2: update the contributors
        update_contributors(version)
    except MakeReleaseException as err:
        print('Could not update the contributor list:\n  ', err)
    # Step 3: update the spec file
    update_spec(version)


if __name__ == '__main__':
    args = parse_args()
    try:
        main(
            version=args.version,
        )
    except MakeReleaseException as err:
        print('ERROR: ', err)

pingou commented 18 days ago

One thing that script does it also updating the contributors list in the
documentation (which I would then build and publish in the doc's website after
releasing)

wombelix commented 18 days ago

[x] Tag last commit in 5.14.x branch with 5.14.1 and publish release
git tag -a 5.14.1 6b06ac5 -m "Release 5.14.1" git push upstream 5.14.1

FYI, I used/use to gpg sign the tags.

OK I guess I should've asked then that you tag the release, sorry about that.

We have to publish very likely a 5.14.2 this week because of some problems on EL8. What do you suggest? I push to the 5.14.x branch and ping you when it's ready to be tagged?

Also, I have a script that I used to make release:
~/bin/make-release.py -v 5.13.3

Thanks for sharing, it looks like I did all that stuff manually what you've put into the script.

version bumps: https://pagure.io/fork/wombelix/pagure/c/937fa7f33d8a6e2322539f04609ace869b8a8c96

contributors list: https://pagure.io/fork/wombelix/pagure/c/18b0586b3e18149c9c183a5dba9f214898dee91d

But great time safer for the next release then :)

One thing that script does it also updating the contributors list in the
documentation (which I would then build and publish in the doc's website after
releasing)

OK so the files are updated in the 5.14.x branch and 5.14.1 tag. Is it somewhere documented how to build and publish the docs? Or is it something you can do?

pingou commented 18 days ago

On Mon, May 27, 2024 at 08:08:42AM +0000, Dominik Wombacher wrote:

wombelix added a new comment to an issue you are following:
``

[x] Tag last commit in 5.14.x branch with 5.14.1 and publish release
git tag -a 5.14.1 6b06ac5 -m "Release 5.14.1" git push upstream 5.14.1

FYI, I used/use to gpg sign the tags.

OK I guess I should've asked then that you tag the release, sorry about that.

We have to publish very likely a 5.14.2 this week because of some problems on EL8. What do you suggest? I push to the 5.14.x branch and ping you when it's ready to be tagged?

Oh I didn't mean to imply I should do the tagging, more like a good practice to
sign tags :)

Also, I have a script that I used to make release:
~/bin/make-release.py -v 5.13.3

Thanks for sharing, it looks like I did all that stuff manually what you've put into the script.

version bumps: https://pagure.io/fork/wombelix/pagure/c/937fa7f33d8a6e2322539f04609ace869b8a8c96

contributors list: https://pagure.io/fork/wombelix/pagure/c/18b0586b3e18149c9c183a5dba9f214898dee91d

But great time safer for the next release then :)

One thing that script does it also updating the contributors list in the
documentation (which I would then build and publish in the doc's website after
releasing)

OK so the files are updated in the 5.14.x branch and 5.14.1 tag. Is it somewhere documented how to build and publish the docs? Or is it something you can do?

I can do it no problem, I'll document the process here while doing it :)

pagure

#5473 Upcoming 5.14 release

Opened a month ago by wombelix. Modified 18 days ago

Metadata

Related Pull Requests

pagure

Source Code

Documentation

#5473 Upcoming 5.14 release Opened a month ago by wombelix. Modified 18 days ago

Close issue as:

Metadata

Related Pull Requests

#5473 Upcoming 5.14 release

Opened a month ago by wombelix. Modified 18 days ago