#458 submit koji to pypi
Closed: Fixed 2 years ago Opened 2 years ago by kparal.

Koji is a dependency for libtaskotron, and we'd like to be able install it from pypi. Can you please submit it to pypi?


I expect, that you're using only library part (aka import koji) and API calls. Would it be enough for you to have this in PyPi? I don't think it would be easy to distribute everything we have in koji (builder, hub, koijra, ...) What about CLI, python 2 or 3 or both? We can add more later, but trying to limit work for now.

Yes, we're using just the koji library (import koji), nothing else. No CLI needed. I only see Python 2 files in the koji package, and Python 2 is all we use right now (of course if you have Python 3 library as well, it's great to submit it as well). Thanks a lot.

I believe we would have to do a bit of repackaging work for this. So, sure, we should do this at some point, but it's not like I can just run the register and upload commands.

I've taken @pbabinca's initial work and tuned it to koji 1.13 and python2/3.

Caveats:

  • rpm is not on PyPi. It needs to be symlinked if used in virtualenv.
  • pycurl is not installed with proper (system) backend by default
  • libcomps is not available via PyPi, which effectively disables import-comps command (but it is rarely used and you probably want full-blown installation (e.g. with plugins) if you are doing such administration tasks)

Any ideas?

I believe you have full access to system binaries when running from virtualenv (just python paths are overridden), so there should be no problem calling rpm, shouldn't it?

What exactly is the problem with pycurl - what are the consequences and how to solve it?

It would be nice to list these issues on the pypi page.

Oh, I mean rpm python bindings, not binary.

I've made a comment about it in PR. Technically it could be compiled with different cryptography backends. pip don't have a way (or I wasn't able to find it) how to enforce used backend. So, you can get message ala 'you are using pycurl without nss against c library with nss', so simply https connections will not work.

For Fedora it is fixed by:

pip uninstall pycurl; pip install pycurl --global-option="--with-nss"

Yeah, documentation of that is definitely necessary.

I see. Let's start with this, and we can try to fine tune in the future. Just having koji in pypi, even when not fully functional, helps with generating documentation on readthedocs-like services, which is already an improvement for us.

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.14

2 years ago

I've tried it on test PyPi. Can you confirm, that it works well enough for you?

Testing e.g. via:

pip install -i https://testpypi.python.org/pypi --extra-index-url https://pypi.python.org/simple koji

Sorry for late reply. I tested the package and it seems to work fine for us. Can you push to proper pypi? Thanks.

@breilly Could you help check if we can push to proper pypi while Tomas is injured?

Hello,

rpm is not on PyPi. It needs to be symlinked if used in virtualenv.
pycurl is not installed with proper (system) backend by default
libcomps is not available via PyPi, which effectively disables import-comps command (but it is rarely used and you probably want full-blown installation (e.g. with plugins) if you are doing such administration tasks)

Any ideas?

I have a solution.

rpm is not on PyPi. It needs to be symlinked if used in virtualenv.

I implemented a installer to install rpm from PyPI, because another project had same kind of situation.
Now it is possible to install rpm from PyPI.
https://github.com/junaruga/rpm-py-installer

libcomps is not available via PyPi, which effectively disables import-comps command (but it is rarely used and you probably want full-blown installation (e.g. with plugins) if you are doing such administration tasks)

I can try to implement the installer too, because it looks same kind of logic with rpm Python binding's installer, if you need it.
Related page: https://github.com/junaruga/rpm-py-installer/issues/67

I came from below thread.
https://pagure.io/fm-orchestrator/issue/700#comment-466400
module-build-service wants to install from pip. and it depends on koji.

I contacted a reporter from below thread.
https://lists.fedoraproject.org/archives/list/koji-devel@lists.fedorahosted.org/thread/XLIR4DSXXRU3OYWXEZWJQJAEEIOUQEXY/?sort=date

He introduced tkopecek (this thread's reporter) is working for that.
https://pagure.io/fork/tkopecek/koji/branch/pypi

rpm is not on PyPi. It needs to be symlinked if used in virtualenv.

Seems that virtualenv /tmp/venv2 --system-site-packages makes site packages accessible in virtualenv. Are there any reasons to avoid using this option?

[0] $ python -mrpm
/usr/bin/python: No module named rpm.__main__; 'rpm' is a package and cannot be directly executed

[0] $ virtualenv /tmp/venv1 
New python executable in /tmp/venv1/bin/python2
Also creating executable in /tmp/venv1/bin/python
Installing setuptools, pip, wheel...done.

[0] $ virtualenv /tmp/venv2 --system-site-packages
New python executable in /tmp/venv2/bin/python2
Also creating executable in /tmp/venv2/bin/python
Installing setuptools, pip, wheel...done.

no rpm module in /tmp/venv1:

[0] $ . /tmp/venv1/bin/activate

[0] $ python -mrpm
/tmp/venv1/bin/python: No module named rpm

[0] $ deactivate

There is one available in /tmp/venv2:

[0] $ . /tmp/venv2/bin/activate

[0] $ python -mrpm
/tmp/venv2/bin/python: No module named rpm.__main__; 'rpm' is a package and cannot be directly executed

Metadata Update from @mikem:
- Issue set to the milestone: 1.15 (was: 1.14)

2 years ago

Currently it seems that we have three options. These two are 'contained' in PR:
1) symlink rpm directory manually to venv
2) use --system-site-packages - from koji's side there will be no harm, but I can imagine a lot of situation where users will don't want to do this from reasons outside of koji itself.
3) use @jaruga 's library - which needs rpm-devel to be installed, so it is also manual step and this one needs root access

@jaruga Is there some optional use of the library? So if there succeeds it is ok and if not, just print some warning and let installation continue? If not, I would stay with current PR as step 1 and 2 are doable by anybody. Or do you have some advice?

@tkopecek

@jaruga Is there some optional use of the library?

Yes, my library has some options.

https://github.com/junaruga/rpm-py-installer/blob/master/docs/users_guide.md#environment-variables

But unfortunately right now rpm-devel is required.

So if there succeeds it is ok and if not, just print some warning and let installation continue? If not, I would stay with current PR as step 1 and 2 are doable by anybody. Or do you have some advice?

Sure. Thanks for the feedback.
I can implement your suggested logic, removing rpm-devel dependency.

After implementing and releasing, let you know in this thread just in case.
Of course whether using my library is optional and your choice.

Created the issue page.
https://github.com/junaruga/rpm-py-installer/issues/70

I wonder if the koji client should be packaged separately from the koji server. Grepping around, I don't see anything in the client that uses the rpm python library.

@ktdreyer Basic library (koji/__init__.py) is using it mainly in get_header_fields function which is further used in few places in CLI.

3) use @jaruga 's library - which needs rpm-devel to be installed, so it is also manual step and this one needs root access

The library that is a installer for RPM Python binding was success to remove rpm-devel dependency from the installer. (not released to pypi yet).

Now I am removing popt-devel dependency of the installer.
popt-devel is a dependency of rpm-devel. And a last dependency to remove. Maybe.

@tkopecek

Now I released rpm-py-installer new version v0.5.0 that does not need rpm-devel (and popt-devel too).
https://pypi.python.org/pypi/rpm-py-installer

Now it "Is there some optional use of the library".

We need to install "dnf download plugin(dnf-plugins-core)" instead of that, if it is not installed on your environment.

$ dnf install 'dnf-command(download)'

or

$ dnf install dnf-plugins-core

if not, just print some warning and let installation continue.

If rpm-devel has been installed on the environment, that is used, and installer is run as previous version's logic.

Right now there is a restriction for the install without rpm-devel library.

  1. Error on Fedora rawhide
    The reason of the error is that DNF download plugin's bug. I reported here.
    https://bugzilla.redhat.com/show_bug.cgi?id=1498426
    If you will install with installed rpm-devel library on Fedora rawhide, it works.

  2. The installer does not work on CentOS6 rpm v4.8.0 environment.
    rpm v4.8.0 was released on January 2010, it is a very old version.
    The reason of the error is the difference of latest and the old version's internal structure.
    I am trying to fix in this page.
    https://github.com/junaruga/rpm-py-installer/issues/86

If you will install with installed rpm-devel library, it might work.

I tested on Fedora 25, 26, rawhide, CentOS7, py27, py34, py35, py36, py37(3.7.0a1).

Error on Fedora rawhide
The reason of the error is that DNF download plugin's bug. I reported here.
https://bugzilla.redhat.com/show_bug.cgi?id=1498426
If you will install with installed rpm-devel library on Fedora rawhide, it works.

Above issue was fixed.

Now I released rpm-py-installer new version v0.5.0 that does not need rpm-devel (and popt-devel too).
https://pypi.python.org/pypi/rpm-py-installer

Guys, if you have something to request for rpm-py-installer, please let me know.

@tkopecek Are you satisfied for the released new version of rpm-py-installer filling your requirement (installing without rpm-devel)?
Are you ready to upload Koji to PyPI?
Do you have any other requirement for Koji in PyPI?

Commit c78a531 relates to this ticket

Commit db7b4c0 relates to this ticket

Metadata Update from @tkopecek:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Related PR (drop pycurl dependency) #775

Login to comment on this ticket.

Metadata