Downstream users (e.g. fedpkg) would then just define new service or plugin for their custom functionality?

The separate branch essentially means this would be version 2 with no backwards compatibility.

Some further requirements:

• it must be possible to have multiple downstream wrappers installed at the same time on one system.
• it must be possible for downstream wrapper to customize behaviour of pyrpkg upstream. For example mapping dist-git branch to koji target.

@pingou you may be interested in this one. AFAIK in the past you mentioned you have some vision how rpkg should look.

The ideas proposed here sound good, my main concern remains around the idea of keep rpkg a CLI tool. Making rpkg a CLI (as opposed to a library) might lead to a situation as we have now where fedpkg tag -l cannot be tested because it's calling rpkg and rpkg prints the results.

We could keep rpk being a CLI tool, but then there needs to be a really clear cut between backend (library) and frontend (the CLI modules), so fedpkg tag -l could do either:

• call rpkg tag -l
• call (say) pyrpkg.git.list_tags() and do the printing itself

This is basically what I had in mind, making sure the logic and the place where the output is generated are clearly distinguished, allowing to use one or the other depending on the need/use-case.

Does that make sense?

Downstream users (e.g. fedpkg) would then just define new service or plugin for their custom functionality?

Yes. Just like the ones defined in rpkg.

The separate branch essentially means this would be version 2 with no backwards compatibility.

Yup, no backwards compatibility. But, how about fork to a new project and keep development happening on that project? That would be easier to maintain, and even the release because at least, before rpkg gets retired thoroughly, no need to make releases for rpkg and new rpkg by switching branches back and forth.

Some further requirements:

it must be possible to have multiple downstream wrappers installed at the same time on one system.
it must be possible for downstream wrapper to customize behaviour of pyrpkg upstream. For example mapping dist-git branch to koji target.

Thanks for mentioning these two items. Both of them should be supported. BTW, is Commands.load_target the one mapping dist-git branch to koji target?

We could keep rpk being a CLI tool, but then there needs to be a really clear cut between backend (library) and frontend (the CLI modules), so fedpkg tag -l could do either:

+1

@pingou I think I get your point. I'm curious that, is anybody using rpkg CLI now, what is the original intention of making rpkg be used as a CLI?

Should be compatible with Python 3

I'm curious that, is anybody using rpkg CLI now, what is the original intention of making rpkg be used as a CLI?

I don't think anyone uses rpkg CLI directly, I believe the derivatives are kind of inheriting the commands/args from rpkg, kinda like extending rpkg's CLI. I think that was sort of the original intention, reduce code duplication by sharing some level of CLI that is then extended in each client, but then the code didn't separate the front-end with the back-end in a good way.

no one uses the rpkg cli directly. The idea was that rpkg would provide base functionality and if that met your needs you would just inherit it as is and things would work. if you needed to do something different then you have to overload things and extend it to meet your needs. I do not think that rpkg needs to have a cli. but it might be nice to provide a base cli that can be a template for others to build upon. I do not think that it is 100% essential as at the least there is fedpkg, centpkg and whatever rpmfusion named their version. making rpkg just be a common library I think is fine

Indeed. When I started automating RPM packaging I wanted to reuse and contribute to fedpkg/rhpkg/rpkg but I was completely discouraged by its codebase.

Thus I started rdopkg which already does all the things required in this proposal and also created pwnpkg manifest which calls to extract this common functionality into a separate module and rewrite fedpkg.

I improve and maintain rdopkg for over 3 years and it's been used by packagers to maintain their RPM packages (most RDO/RHOSP packages plus some non-RDO ones like Ceph) but it's also being extensively used in CI and automation (DLRN, RDO CI, RHOSP CI). So, instead of splitting rdopkg into a separate RPM packaging framework (pwnpkg), I decided to make it reusable and focused in one place - rdopkg. I already finished disruptive refactorings required for that.

In other words, I'm commited in making rdopkg the ultimate RPM packaging automation tool AND framework and move the entire toolchain forward.

I suggest rewriting fedpkg using rdopkg because:

• rdopkg enables easy creation of modular, reusable, maintainable and lightweight CLIs through plugins (action modules) that define both command line interface and module entry points in one place without redundancy. For example, see how easy it is to create new CLI commands to expose spec parser in new plugin. Modules are only loaded on-demand as opposed to importing every module you could possibly need on each run (cheers to openstackclient). Transactions are also supported and allow multi-step workflow a la git rebase --continue
• rdopkg itselfs strives to be clean, reusable, extendable and maintainable. That seems successfull to a point whererdopkg users tend to contribute back to rdopkg on their own.
• rdopkg focuses on clean interfaces, both CLI and python ones.
• rdopkg already has much of the code needed for RPM packaging such as .spec parser/editor, koji/copr interface, package env detection and much more
• rdopkg has CI and increasing collection of unit and integration tests that are run against every change.
• rdopkg has dedicated and responsive maintainer with a vision for years to come, happy to help new users and contributors alike.

So please, do contact me (jruzicka @ #rdo @ freenode IRC) if you want to make nice fedpkg happen, I think we all deserve it. I don't have time to do the rewrite myself but I'll provide maximum support including initial code using rdopkg (that should be only a few lines of code, really - that's the point) and reviews. Still, we need someone to focus on this.

Power to the packagers!

Cheers,
Jakub

I created a rdopkg issue to track progress of support for creating packaging CLIs using rdopkg.

Thanks for sharing your thoughts. Actually, we have similar ideas of implementation of rdopkg and the proposed next generation of rpkg. Recently, we, current rpkg maintainers, have a nice talk about the future well-redesigned and refactored rpkg. By taking this opportunity of replying your comment, let me share some thoughts came from our talk.

As you know that, rpkg codebase has limitation and disadvantages for current maintenance and make it not easy to add new features to it. Everything goes into one class either the cliClient or pyrpkgClient, as time goes by, they is becoming bigger and bigger. This could cause problems, that different developers could see different issues from it.

What can we do to make rpkg better, then to benefit devleopment of downstream package tools like fedpkg and rhpkg? That is to split and modularize current rpkg into several dedicated components and use "plugin" mechanism to organize subcommands, such as clone, commit, mockbuild, and update. Okay, more details, new rpkg would be

• a framework of CLI to allow developers to create CLI application easily without copy and paste.

• a set of components (or they could be called libraries, modularize modules, whatever the name) that can be used to get various information from package and its git repository and interact with backend package infrastructure like dist-git and lookaside. They may include

  • PackageRepo: interface to get underlying git metadata, e.g. tags, branches

  • Package: interface to get package information (the metadata), e.g. module name, NVR, disttag, distval, distvar, epoch, giturl.

  • Distgit: interface to operate package repository, e.g. clone, commit, push

  • Koji: interface to interact with Koji to build packages, e.g. scratch-build, build, mock-config

  • Lookaside: interface to upload and download sources files

  • LocalBuild: interface to build package locally

  all these could be reuse by any other applications.

• a framework to allow developers to write new commands for their package tool easily. A new command can be written, distributed and installed outside rpkg as long as it follows some rules defined by rpkg so that rpkg is able to discover and load it dynamically.

  A little more details include

  • rpkg provides a set of core commands dedicated to package build and operations with Lookaside, which are good for sharing with downstream package tools.

  • downstream package tool developers can write their own commands, and those commands can be totally new, a customized based on core commands, or disabled.

  Currently, this would benefit fedpkg and rhpkg at least.

That's it I think. @lsedlar, feel free to add any new thoughts and anything that I missed here.

I had a quick look at rdopkg. With above changes, for example, rdopkg is able to build package without making subprocess to execute fedpkg, instead,

from rpkg import koji
koji.build(...)

I hope I can start continue to write some code recently to implement all the thoguths step by step, and then anyone who is interested in can have a try.

Finally, regarding the development, I will write new code in a separate repository and move existing commands, the core commands from new rpkg perspective, one by one from original rpkg.

Thanks for sharing your thoughts. Actually, we have similar ideas of implementation of rdopkg and the proposed next generation of rpkg. Recently, we, current rpkg maintainers, have a nice talk about the future well-redesigned and refactored rpkg. By taking this opportunity of replying your comment, let me share some thoughts came from our talk.

I've been working on rpkg replacement past 3 years - it's called rdopkg
and after much development and refactoring it's ready to be used as
a framework to build fedpkg2, rhpkg2, copr-cli, centospkg and all other
such tools while reusing common code. It has growing userbase, unit
tests, documentation, CI, open review pipeline and more.

As you know that, rpkg codebase has limitation and disadvantages for current maintenance and make it not easy to add new features to it. Everything goes into one class either the cliClient or pyrpkgClient, as time goes by, they is becoming bigger and bigger. This could cause problems, that different developers could see different issues from it.

These limitations are addressed in rdopkg. It has modular lightweight
plugin architecture that allows easily maintaining many different
commands under one tool while encouraging single clear interface for CLI
and other python modules. It even allows for basic transaction that
allow drop to shell and then continue a la git rebase --continue.

What can we do to make rpkg better, then to benefit devleopment of downstream package tools like fedpkg and rhpkg? That is to split and modularize current rpkg into several dedicated components and use "plugin" mechanism to organize subcommands, such as clone, commit, mockbuild, and update. Okay, more details, new rpkg would be

That's what rdopkg action modules are. Lightweight python plugins that
have interface declaration in init.py:

https://github.com/openstack-packages/rdopkg/blob/master/rdopkg/actions/info/__init__.py

which maps to action functions in actions.py:

https://github.com/openstack-packages/rdopkg/blob/master/rdopkg/actions/info/actions.py

Only init.py is needed to construct CLI and help, actual code is
loaded on-demand as opposed to LOADING ALL THE MODULES.

Please see how simple is it to add a new action module that adds
CLI commands with options. Minimal redundancy, just add a module:

https://review.rdoproject.org/r/#/c/5613

• a framework of CLI to allow developers to create CLI application easily without copy and paste.

That's exactly what rdopkg is. Look how simple is the rdopkg main()
(after a refactor):

https://github.com/openstack-packages/rdopkg/blob/master/rdopkg/cli.py

Creating new tool with rdopkg framework is equally easy. Only few small
things like replacing rdopkg with $YOURTOOLNAME remain for this to be
officially supported, but I can finish this for next release if needed.

Initially I wanted to create a framework just for pluggable CLI creation
but that was too much of splitting effort so it's included in rdopkg.

I updated pwnpkg manifest to reflect current rdopkg as a framework
state:

https://github.com/yac/pwnpkg

• a set of components (or they could be called libraries, modularize modules, whatever the name) that can be used to get various information from package and its git repository and interact with backend package infrastructure like dist-git and lookaside. They may include

  • PackageRepo: interface to get underlying git metadata, e.g. tags, branches

  • Package: interface to get package information (the metadata), e.g. module name, NVR, disttag, distval, distvar, epoch, giturl.

  • Distgit: interface to operate package repository, e.g. clone, commit, push

  • Koji: interface to interact with Koji to build packages, e.g. scratch-build, build, mock-config

  • Lookaside: interface to upload and download sources files

  • LocalBuild: interface to build package locally

  all these could be reuse by any other applications.

Yes, these are called "action modules" in rdopkg, you can see all the
rdopkg functionality split into action modules here:

https://github.com/openstack-packages/rdopkg/tree/master/rdopkg/actions

When reusing rdopkg, you can use your own actions or reuse the
rdopkg ones if convenient. In case of fedpkg2, one would move fedpkg
code to rdopkg action modules in a separate tool and repo.

• a framework to allow developers to write new commands for their package tool easily. A new command can be written, distributed and installed outside rpkg as long as it follows some rules defined by rpkg so that rpkg is able to discover and load it dynamically.

  A little more details include

  • rpkg provides a set of core commands dedicated to package build and operations with Lookaside, which are good for sharing with downstream package tools.

  • downstream package tool developers can write their own commands, and those commands can be totally new, a customized based on core commands, or disabled.

  Currently, this would benefit fedpkg and rhpkg at least.

That's it I think. @lsedlar, feel free to add any new thoughts and anything that I missed here.

I had a quick look at rdopkg. With above changes, for example, rdopkg is able to build package without making subprocess to execute fedpkg, instead,

from rpkg import koji koji.build(...)

I hope I can start continue to write some code recently to implement all the thoguths step by step, and then anyone who is interested in can have a try.

Before writing your own from scratch, please try looking at reusing
rdopkg first. I created it exactly for this purpose. I just don't have
manpower to both maintain rdopkg AND rewrite fedpkg/rhpkg... if you
have time/will to do that, I can save you a lot of time :)

Finally, regarding the development, I will write new code in a separate repository and move existing commands, the core commands from new rpkg perspective, one by one from original rpkg.

Yes, splitting code into modules in a new repository is required.

Compared to starting from scratch, rdopkg already solves many
CLI/plugin/logging/shell issues while providing handy tools like
shell/git interface, spec parser/editor and what. You don't need
to maintain that because I will, just focus on the actual fedpkg/rhpkg
functionality.

It will be best to demonstrate with code - let me create fedpkg2
repository using rdopkg and see if you like it.

rpkg should be a generic Python package to build package tools that builds package with dist-git and Koji, it must be as much general as possible. I don't want rpkg, fedpkg and rhpkg to be built based on a codebase that includes any other irrelative code.

It looks rdopkg does something specific to RDO packaging, and build RPMs in Fedora by executing fedpkg CLI directly.

Ideally, the new architecture of package tools around new rpkg should look like this

                 +------+
     +-----------> rpkg <-----------+
     |           +---^--+           |
     |               |              |
     |               |              |
+----+---+       +---+---+      +---+----+
| fedpkg |       | rhpkg |      | rdopkg |
+--------+       +-------+      +--------+

rather than

   +-------------------+
   |         +------+  |
   |  rdopkg | rpkg |  |
   |         +------+  |
   +------^-----^------+
          |     |
          |     |
    +-----+     +----+
    |                |
+---+----+       +---+---+
| fedpkg |       | rhpkg |
+--------+       +-------+

Note: line from rdopkg to rpkg means rdopkg reuses rpkg to build packages in Fedora.

If you can move code, which is only relative to make packages with Koji and dist-git, into a separate package, that would be helpful. Basically, that is what the new rpkg does I mentioned above.

Yes, these are called "action modules" in rdopkg, you can see all the
rdopkg functionality split into action modules here:

I don't quite understand what "action" mean and what is the definition of a "action" in rdopkg? However, from literal meaning of word action, it is, in a way, relative to the third item I mentioned, that is "a framework to allow developers to write new commands for their package tool easily".

Can actions now do these? All of them are required.

• allow developer to write new command in a standalone Python package, that means the new command could have its own code repository, be distributed and packaged in its own package, and be installed into its own location.
• discover installed new command automatically.
• allow to disable or customize an existing command provided by upstream. For example, in XXpkg,
  • to disable command X provided by rpkg
  • add additional arguments to command Y

BTW, what is the relationship between modules actionmods and actions?

rpkg should be a generic Python package to build package tools that builds package with dist-git and Koji, it must be as much general as possible. I don't want rpkg, fedpkg and rhpkg to be built based on a codebase that includes any other irrelative code.
It looks rdopkg does something specific to RDO packaging, and build RPMs in Fedora by executing fedpkg CLI directly.

A fair requirement. rdopkg is indeed too many things at once.

I initially wanted to split into 3 parts:

• pwncli - a generic pluggable CLI creation framework
• pwnpkg - a set of generic RPM packaging related modules
• rdopkg - RDO specific packaging tool

...but time.

Ideally, the new architecture of package tools around new rpkg should look like this
+------+
+-----------> rpkg <-----------+
| +---^--+ |
| | |
| | |
+----+---+ +---+---+ +---+----+
| fedpkg | | rhpkg | | rdopkg |
+--------+ +-------+ +--------+

Agreed. In fact, when I started rdopkg I wanted to use rpkg like that and
others did too so making that usecase viable is a great goal. I'm happy to
make rdopkg use rpkg if it will provide comparable functionality and code
quality.

I'd also gladly help, is the new rpkg going to use open review-based
development process which I can influence? As opposed to, you know, git push :)

rather than
+-------------------+
| +------+ |
| rdopkg | rpkg | |
| +------+ |
+------^-----^------+
| |
| |
+-----+ +----+
| |
+---+----+ +---+---+
| fedpkg | | rhpkg |
+--------+ +-------+

Indeed.

So by reusing rdopkg, we're talking about extracting the useful codez and/or
ideas from it into a seprate project in this case.

Note: line from rdopkg to rpkg means rdopkg reuses rpkg to build packages in Fedora.
If you can move code, which is only relative to make packages with Koji and dist-git, into a separate package, that would be helpful. Basically, that is what the new rpkg does I mentioned above.

Yup, let's move(/create) all the generic code into a separate project for easy
packaging CLI creation. rpkg2, pyrpkg-ng, pwnpkg... I don't really care how
it's called.

Yes, these are called "action modules" in rdopkg, you can see all the
rdopkg functionality split into action modules here:

I don't quite understand what "action" mean and what is the definition of a "action" in rdopkg? However, from literal meaning of word action, it is, in a way, relative to the third item I mentioned, that is "a framework to allow developers to write new commands for their package tool easily".

Action is basically a CLI subcommand and related codes. For example
new-version action can be seen in rdopkg new-version. It could be called
command... but that's maybe too generic.

Can actions now do these? All of them are required.

allow developer to write new command in a standalone Python package, that means the new command could have its own code repository, be distributed and packaged in its own package, and be installed into its own location.

Yes.

discover installed new command automatically.

Yes.

allow to disable or customize an existing command provided by upstream. For example, in XXpkg,

Collecting and combining actions from different modules is supported.

Replacing isn't but it should be a matter of one patch.

to disable command X provided by rpkg
add additional arguments to command Y

This is also not supported yet, but I thought of it and architecture allows
both to be added in a patch or two.

While the actions handling code is arguably shady (action.py) and I wouldn't
mind someone improving/rewriting it, I think the plugin format I prototyped
with action modules is simple yet powerful.

If you can iterate and make it even better... yay!

BTW, what is the relationship between modules actionmods and actions?

Good question. It's a confusing legacy. actions.py was initally a single file
with interface for entire project. That wasn't modular as you can imagine so
I refactored the interface into action modules but the functionality remained
in previous actionmods location due to backward compat. Now, you would include
these in action modules alongside init.py and actions.py.

An example action module could look like this:

koji
├── __init__.py  <- interface declaration
├── actions.py   <- top level entry poitns (actions)
└── koji.pyc     <- actual reusable code in one or more modules

Once placing such action module into actions directory, all contained actions
(CLI subcommands) are available.

So a new rpkg project is a good idea and I'd like to help with making sure
that rpkg2 >= rdopkg - iterate on rdopkg to reuse and expose all the useful
bits while learning and improving from mistakes of both rpkg and rdopkg.

I'll try to write down what is cool about rdopkg and what should be reused but
also what sucks and howto not repeat my mistakes, on both code and idea
levels.

Until then, please inspect the
distgit action module,
especially the
interface declaration in init.py.

Finally, is there some IRC or other text channel where one can interact with
rpkg devs?

The separate branch essentially means this would be version 2 with no backwards compatibility.

Yup, no backwards compatibility. But, how about fork to a new project and keep development happening on that project? That would be easier to maintain, and even the release because at least, before rpkg gets retired thoroughly, no need to make releases for rpkg and new rpkg by switching branches back and forth.

A new project definitely needs to be created in order for smooth
transition. Both versions need to coexist until no software uses the old
one.

That brings us to the hardest problem in informatics - the naming!

Please let's settle on a new name of the rpkg replacement. Ideally the same
name for the repository, python module, RPM, and PyPI project. I reckon least
surprising name would be rpkg2.

Once settled, someone can create the project. I assume Pagure will be used
which includes a bug tracker and some docs intergration (I wish it showed
README.md by default, though) so we can start referencing that location and
accumulating ideas/specs/docs there.

Finally, is there some IRC or other text channel where one can interact with rpkg devs?

I'm interested in this as well.

Finally, is there some IRC or other text channel where one can interact with rpkg devs?

I'm interested in this as well.

So far, I actually don't know if there is such an IRC channel. I usually talk with co-maintainers and Fedora guys about rpkg in several irc channels, like #fedora-devel and #fedora-admin.

I would prefer to have an IRC channel and mailing list dedicated for package tool development specifically.

/cc @ausil @lsedlar @pbabinca what do you think about this?

@jruzicka

Lots of thoughts we have so far. Before writing code, I will summary thoughts into a document in next a few days, so that we can share and exchange ideas based on that. Hopefully, I could finish it in next week before my vocation. A little busy these days. As you said, "... but time" :)

Summary is documented at https://fedoraproject.org/wiki/Thoughts_on_rpkg2

Thanks cqi for the update. I see there is a rpkg2 repo on pagure.io as well.

I really hope we can reuse as much of rdopkg as possible. There are some amazing features there.

This issue has been unresolved for more than a year, and is going to be closed within a week if no further action is taken. If you feel this is in error, please contact me.
This is a cleaning process suggested by Jay Greguske. Copy of this ticket was already closed in JIRA tracker.

Feel free to close this issue. The development of refactored rpkg is in rpkg2. The development is slow and stalled for now. And any future updates will be done there.