#141 Add documentation about packit involvements
Merged a year ago by zlopez. Opened a year ago by jrichardson.
fedora-infra/ jrichardson/arc main  into  main

@@ -59,6 +59,7 @@ 

      monitorgating

      releng

      packagers

+     packit

  

  Index

  -------

@@ -0,0 +1,127 @@ 

+ .. _packit:

+ 

+ Pagure Dist Git Interactions With Packit

+ ========================================

+ 

+ Packit is a CLI tool that helps developers auto-package upstream projects

+ into Fedora operating system. You can use packit to continuously build your 

+ upstream project in Fedora. With packit you can create SRPMs, open pull 

+ requests in dist-git, submit koji builds and even create bodhi updates, 

+ effectively replacing the whole Fedora packaging workflow.

+ 

+ Packit interacts with Dist Git mainly by making pull requests (eg. with version bumps)

.. and reacting to events around pull-requests (e.g. adding comments).

Also, we react to push events.

If it works more reliably, we would also want to work with issues.

+ 

+ `Packit Source Code <https://github.com/packit/packit/>`_

+ 

+ 

+ Possible Interactions

+ ---------------------

+ 

+ The following are a set of documented possible interactions between Pagure Dist

+ Git and Packit

+ 

+ The ``DistGit`` class

+ ~~~~~~~~~~~~~~~~~~~~~

+ 

+ - Packit has a class named 

+   `DistGit <https://github.com/packit/packit/blob/main/packit/distgit.py>`_

+   that interacts with dist-git and pagure-over-dist-git API. 

+   The logic covers git and pagure interaction, manipulation with content,

+   spec files, patches and archives. One optional argument, "clone_path" is the path

+   where the dist-git repo is cloned, this will be affected by a Dist Git deployment

+   change. There are a number of methods in the class that interact with dist-git listed 

+   below.

+ 

+ 

+ - `Cloning the repo <https://github.com/packit/packit/blob/main/packit/distgit.py#L97>`_

+   This method clones the dist-git repo for the selected package and returns 

+   the instance of the Dist Git class. Packit uses a wrapper,

+   `pkgtool <https://github.com/packit/packit/blob/main/packit/pkgtool.py#L13>`_

+   that utilizes `fedpkg clone method <https://github.com/packit/packit/blob/main/packit/pkgtool.py#L136>`_

+   which builds the clone command from getting the fas username, directory, branch, and target path.

+ 

+   - API Endpoint: ``/api/0/<namespace>/<repo>``

+ 

+ - `Cloning the package <https://github.com/packit/packit/blob/main/packit/distgit.py#L180>`_

+   This method clones the package from Dist Git, utilizing the same ``clone`` method in the 

+   ``pkgtool`` wrapper. Optional argument for the branch to checkout, if so, an additional API call

+   is made to the endpoint below.

+ 

+   - API Endpoint: ``/api/0/<namespace>/<package_name>``

+   - API Endpoint: ``/api/0/<namespace>/<package_name>/git/branches``

+ 

+ - `Obtaining absolute path for the specfile <https://github.com/packit/packit/blob/main/packit/distgit.py#L207>`_

+   Determines the absolute path for the specfile in Dist Git. Specfile path is built using the

+   configured ``spec_dir_name``, then ``f"{self.package_config.downstream_package_name}.spec"``

+ 

+   - API Endpoint: ``/api/0/<namespace>/<repo>``

+ 

+ - `Updating branch <https://github.com/packit/packit/blob/main/packit/distgit.py#L233>`_

+   Fetches the latest commits to the selected branch. Needs to hit the same API Endpoint as the

+   package cloning method. Packit has a 

+   `LocalProject class <https://github.com/packit/packit/blob/6e286a7b4d0f79cd2a8213a8ae978788be5219c5/packit/local_project.py#L31>`_

+   That represents the cloned repo, the **fetch** method is defined there

+   `Fetch method <https://github.com/packit/packit/blob/6e286a7b4d0f79cd2a8213a8ae978788be5219c5/packit/local_project.py#L560>`_

+ 

+   - API Endpoint: ``/api/0/<namespace>/<package_name>/git/branches``

+ 

+ - `Synchronizing Access Control Lists <https://github.com/packit/packit/blob/main/packit/distgit.py#L255>`_

+   Synchronizes ACLs between the Dist Git project and its fork. Users and groups 

+   with commit or higher access to the original project will gain commit access 

+   to the fork. Users and groups without commit or higher access to the original 

+   project will be removed from the fork. Utilizes ``ogr.abstract`` methods in the

+   `GitProject class <https://packit.github.io/ogr/abstract.html#ogr.abstract.GitProject>`_

+ 

+   - API Endpoint: ``/api/0/<namespace>/<package_name>/hascommit``

+   - API Endpoint: ``/api/0/<namespace>/<package_name>/git/modifyacls``

I hope the ACL edits won't be needed on other forges -- this is a hack to allow maintainers to edit the Packit's pull-requests on Pagure. (What works out-of-the-box on other forges.)

+ 

+ - `Pushing to a fork repository <https://github.com/packit/packit/blob/main/packit/distgit.py#L282>`_

+   Push changes to a fork of the Dist Git repo after checking if user has access. Hits the same API Endpoint

+   as **Cloning the repo** above. 

+ 

+   - API Endpoint: ``/api/0/<namespace>/<package_name>``

+ 

+ - `Creating pull request <https://github.com/packit/packit/blob/main/packit/distgit.py#L328>`_

+   Creates a Dist Git PR using the requested branches

+ 

+   - API Endpoint: ``/api/0/<namespace>/<package_name>/pull-request/new``

+ 

+ - `Downloading upstream tarballs <https://github.com/packit/packit/blob/main/packit/distgit.py#L382>`_

+   Fetch archives for the current upstream release defined in Dist Git's spec. Again, uses the fedpkg

+   wrapper ``packit/pkgtool.py``, specifically the

+   `sources method in pkgtool.py <https://github.com/packit/packit/blob/main/packit/pkgtool.py#L64>`_

+ 

+ - `Downloading source files <https://github.com/packit/packit/blob/main/packit/distgit.py#L403>`_

+   Downloads source files from the lookaside cache. Using the the **sources** method in 

+   ``packit/pkgtool.py``

+ 

+ - `Uploading the source files to the lookaside cache <https://github.com/packit/packit/blob/main/packit/distgit.py#L420>`_

+   Uploads files (archives) to the lookaside cache. Uses the 

+   `new_sources method in pkgtool.py <https://github.com/packit/packit/blob/main/packit/pkgtool.py#L42>`_

+   method in ``packit/pkgtool.py``

+ 

+ - `Checking if lookaside cache is already populated <https://github.com/packit/packit/blob/main/packit/distgit.py#L456>`_

+   Checks whether the archive is already uploaded to the lookaside cache, using a wrapper around lookaside cache

+   `lookaside.py <https://github.com/packit/packit/blob/main/packit/utils/lookaside.py#L17>`_ This returns

+   True if the archive is present in the lookaside cache, False if not.

+ 

+ Changes proposed

+ ----------------

+ 

+ In order to switch out **Pagure Dist Git** for an alternative that provides a

+ relatively compatible API pattern, the following changes must be made in the

+ places mentioned below.

+ 

+ Necessary

The following two points are valid. We either use fedpkg and some code on top of it or our OGR library that should take care of the interaction with git-forge.

The last bit is the processing of git-forge events. We either need to let people set webhooks for dist-git projects (or define global ones). Or, ideally, we need to have the same messages on the fedora-messaging bus as we have now.

+ ~~~~~~~~~

+ 

+ - Packit uses a number of URLs that point to dist-git mentioned above. If the URL should

+   change, they would need to be updated.

+ 

+ - Packit team has created a library that provides the same API for various git forges,

+   so the proposed changes above would be necessary for OGR if another git forge is used.

+   `Packit/ogr <https://github.com/packit/ogr>`_

+ 

+ Unnecessary

+ ~~~~~~~~~~~

+ - none

1 new commit added

  • Add documentation about packit involvements
a year ago

rebased onto fc041dbc1a39846114171c7ce76dd85511ecaa8d

a year ago

Could you please go into detail regarding the various methods in the class about actions like cloning the repository (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L97), cloning the package (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L180), obtaining absolute path for the specfile (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L207), updating branch (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L233), synchronizing ACLs (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L255), pushing to a fork repository (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L282), creating a pull request (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L328), downloading upstream tarballs (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L381), downloading source files (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L402), uploading the source files to the lookaside cache (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L419), checking if lookaside cache is already populated (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L455), removing unused Git branches (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L471), obtaining latest build for a branch (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L499), obtaining changelog since last stable build (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L535), creating update on Bodhi (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L571), obtaining allowed GPG keys from downstream configuration (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L676), obtaining associated username (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L689) and obtaining Bugzilla IDs from changelog (ref. https://github.com/packit/packit/blob/main/packit/distgit.py#L693)?

Could you please go into detail explaining the various methods in the class like creating a Dist Git instance object from the URL and namespace (ref. https://github.com/packit/packit/blob/main/packit/dist_git_instance.py/#L24) and checking if the remote URL corresponds to the Dist Git Instance object (ref. https://github.com/packit/packit/blob/main/packit/dist_git_instance.py/#L45)?

OGR currently supports Pagure, GitLab and GitHub so I do think that the changes would be necessary to make to OGR if we end up using a fourth Git forge which is not one of those that OGR supports. Please mention the same.

1 new commit added

  • Incorporate methods within the mentioned classes
a year ago

rebased onto 515c62c02447c020e3319101312427cd1959da80

a year ago

@jrichardson, the changes are still superficial in terms of the depth necessary for this investigation. Could you please explore the definitions and variable usages per method to let us know how each method works and how they would end up changing with a possible alternative put in place of the existing Dist Git deployment? Also, the URL format is mistaken so you would want to change those too while you are it. Please feel free to take a look at the sources of previously done investigations to get an idea about how you should go ahead with this.

rebased onto ceba83af1deca3a29d326c105835d1ddb670b72c

a year ago

Most recent changes and additions. Also reformatted files using black

rebased onto 3aae8dcaa1d65ce256883470e23617eb2ae13ed7

a year ago

rebased onto 3007723

a year ago

@jrichardson It seems you accidentally ran black on everything, could you remove the files that are not related to your changes to different pull request as it's not great to have these changes under same commit.

It would be nice to write here how is that being done. Through API? What API calls?

Still missing the exact method how this is done. Just calling git clone or is there some wrapper?

I'm still missing what API/HTTP endpoints are being called in dist-git. The links to code are nice, but it would be better if there would be just what is being called there.

For example:

`Cloning the repo 

<https://github.com/packit/packit/blob/main/packit/distgit.py#L97>`__

This method clones the dist-git repo for the selected package and returns 
the instance of the Dist Git class. It is calling `fedpkg clone` to do it

Can you please ensure that the links are formatted on the same line?

This might not parse properly in Sphinx.

@jrichardson, you seem to have used two underscores for the link format and it usually is just one underscore at the end. The improper usage of the hyperlink format can potentially lead to the hyperlink not being rendered properly in the generated HTML document. Please correct those.

rebased onto 5954e74

a year ago

@jrichardson It seems you accidentally ran black on everything, could you remove the files that are not related to your changes to different pull request as it's not great to have these changes under same commit.

Removed the reformatted files from this pull request

rebased onto 7cbf2dd

a year ago

Still missing the exact method how this is done. Just calling git clone or is there some wrapper?

It's using a wrapper around fedpkg, included more info on this in the PR

rebased onto bd39bda

a year ago

@jrichardson, you seem to have used two underscores for the link format and it usually is just one underscore at the end. The improper usage of the hyperlink format can potentially lead to the hyperlink not being rendered properly in the generated HTML document. Please correct those.

All links correct to one underscore

rebased onto 4f8784d

a year ago

rebased onto dfae947

a year ago

rebased onto f7bd021

a year ago

rebased onto 044aba6

a year ago

rebased onto 7f9c7e4

a year ago

Just list API calls here again, so it's visible.

Just list API calls here again, so it's visible.

Maybe list the fedpkg sources same way as API endpoints.

rebased onto a673bd2

a year ago

rebased onto deaf9da

a year ago

This shouldn't be here. When I was talking about the fedpkg sources I meant the method itself. But you already have it mentioned above. So just remove these lines.

rebased onto 3a3c69c

a year ago

rebased onto fdf22fe

a year ago

Pull-Request has been merged by zlopez

a year ago

.. and reacting to events around pull-requests (e.g. adding comments).

Also, we react to push events.

If it works more reliably, we would also want to work with issues.

I hope the ACL edits won't be needed on other forges -- this is a hack to allow maintainers to edit the Packit's pull-requests on Pagure. (What works out-of-the-box on other forges.)

The following two points are valid. We either use fedpkg and some code on top of it or our OGR library that should take care of the interaction with git-forge.

The last bit is the processing of git-forge events. We either need to let people set webhooks for dist-git projects (or define global ones). Or, ideally, we need to have the same messages on the fedora-messaging bus as we have now.