#2137 Bodhi 4 to stable releases?
Closed: Accepted 3 months ago by zbyszek. Opened 3 months ago by bowlofeggs.

Greetings!

Bodhi 4 was deployed to production today. It is unfortunately not backwards compatible in a way that breaks that Bodhi 3 client, which is the current stable version in EPEL 7, Fedora 29, and Fedora 30.

So far, the breakage I am aware of is that it cannot render updates. It is able to create them, it seems, but it crashes after creating them while trying to print them to the screen. Thus, it's not highly severe, but it is confusing for packagers.

I have so far made Bodhi 4 available to Rawhide and I have also put it into Copr at bowlofeggs/bodhi.

While I could patch the Bodhi 3 client in the stable releases to be compatible with the Bodhi 4 server, that would then mean that the client in those releases would not be compatible with the server that is also shipped in those releases.

There are some packages that depend on Bodhi's client in Fedora. Some of them use it as a subprocess and some of them import it and use it as a Python library. There were some changes to both usages so we'd have to inspect each one case-by-case to see what they are doing to find out if it is safe or not.

Another complicating factor is that there are at least two other Bodhi deployments in the world, aside from Fedora's, and I cannot easily know if they are relying on Fedora's client or not. The Bodhi 4 client may just as well be incompatible with the Bodhi 3 server, so upgrading could break those deployments as well.

I don't have a proposal at this time. It would be good to gather more information so I can identify which dependencies might be affected if I were to update Bodhi to version 4.

Any suggestions or recommendations from FESCo here?


Would it be possible to patch the bodhi 3 client to be compatible with both versions? Maybe it doesn't have to provide the full informative output, just not crash.

$ dnf repoquery --whatrequires python3-bodhi-client
bodhi-client-0:3.14.0-1.fc30.noarch
bodhi-server-0:3.14.0-1.fc30.noarch
fedora-easy-karma-0:0-0.40.20181206gitb6d97e0.fc30.noarch
fedpkg-0:1.37-1.fc30.noarch
packit-0:0.4.1-1.fc30.noarch
$ dnf repoquery --whatrequires bodhi-client
cmake-fedora-0:2.9.3-4.fc30.noarch
fedora-packager-0:0.6.0.2-5.fc30.noarch

I'm with @zbyszek - we should make the client compatible with both, even if it has a large if.

Would it be possible to patch the bodhi 3 client to be compatible with both versions? Maybe it doesn't have to provide the full informative output, just not crash.

For what it's worth, I'd much rather see the Bodhi 4 client capable of talking to both, that way we can just upgrade everyone everywhere cleanly.

@bowlofeggs This would be a tremendous hack, but could you just do auto-detection of which server you're talking to and then just keep both codebases in the library? Just drop the entirety of the v3 codebase there and fall back to it if you're talking to v3?

AFAIU this broke fedora-easy-karma: https://bugzilla.redhat.com/show_bug.cgi?id=1714925
Not sure if it is still the major tool to submit karma but if it is, there might be big loss of testing feedback now. This could make a lot of people unhappy so a quick solution would be great.

Here's another bug about one of the crashes: https://bugzilla.redhat.com/show_bug.cgi?id=1714950

I considered making the Bodhi 3 client be able to talk to the Bodhi 4 server. It could be done but will take some effort. Making Bodhi 4 talk to Bodhi 3 doesn't seem as good to me because then I have to maintain that code in the long term, and also it would mean upgrading the server to Bodhi 4 in stable Fedoras as well (which is very majorly backwards incompatible).

Another problem is that EPEL 7 actually still has Bodhi 2, and that code is even older. So a patch that works on Bodhi 3 would likely not apply on Bodhi 2…

Users who are hitting this can work around it for now by using my Copr, if they wish: https://copr.fedorainfracloud.org/coprs/bowlofeggs/bodhi

I had a chat with @churchyard in #fedora-admin today, and I think we came up with a reasonable proposal together (most of it his ideas ☺):

  • Create a bodhi3-client package for F29/30. This will only contain the client and not the server, and is available for any users who may be using the client with a Bodhi 3 server. We don't know if there are any Fedora users using Fedora to work with a private Bodhi server, we just know that there are private Bodhi servers out there.
  • Update F29/30 to Bodhi 4 client and server.
  • There was a suggestion to make the Bodhi 4 client be able to detect a Bodhi 3 server and recommend a Bodhi 3 client. In my opinion, this may be more effort than it's worth (requires a new server and client release for Bodhi 4) and I would opt not to do it, but I welcome counter arguments.

Note that this would mean we are not maintaining a Bodhi 3 server anymore for either release, just a client. To be transparent that is mostly just to save effort on our part.

If we can get this approved and implemented, we will make a similar proposal to the EPEL SC for EPEL 7.

we came up with a reasonable proposal together

What happens with packages that depend on python3-bodhi-client and bodhi-client? Will they be updated to the new API in F29/30?

What happens with packages that depend on python3-bodhi-client and bodhi-client? Will they be updated to the new API in F29/30?

They need to be, yes. It doesn't really serve any purpose if they use bodhi3-client and cannot talk to Fedora's Bodhi server.

On Wed, 2019-05-29 at 16:27 +0000, Zbigniew J=C4=99drzejewski-Szmek wrote:

What happens with packages that depend on python3-bodhi-client and
bodhi-client? Will they be updated to the new API in F29/30?

We would need to check each one and see if it should work with bodhi 4
or not. If it doesn't, we can switch it to use the bodhi3 package.

They need to be, yes. It doesn't really serve any purpose if they use bodhi3-client and cannot talk to Fedora's Bodhi server.

We would need to check each one and see if it should work with bodhi 4
or not. If it doesn't, we can switch it to use the bodhi3 package.

Those answers are contradictory ;) I think @churchyard is right here — it would be pointless if they can't talk to our Bodhi server. Until we know if they actually work with the new client, we can't judge the impact.

I was directed here from a thread about an error when running the hourly yum cronjob. The error received is:

Updateinfo file is not valid XML: <open file="" '="" var="" cache="" yum="" x86_64="" 7="" epel="" 92f2e15cad66d79ea1ad327e2af7af89d98e4d153d7a3e27ff41946f476af5b4-updateinfo.xml.zck',="" mode="" 'rt'="" at="" 0x7f29f8719660="">

The system does not have anything related to bodhi installed, so I cannot install a new version of that to fix it.

I have tried 'yum clean all' and have even deleted everything under /var/cache/yum ... but on running the cronjob or "yum check-update" the error still happens. I have seen messages indicating the repository has been fixed ... but that does not seem to be the case for me. Maybe some mirrors aren't yet, and the fastest mirror plugin is choosing one of those?

On Wed, 2019-05-29 at 20:41 +0000, Zbigniew J=C4=99drzejewski-Szmek wrote:

Those answers are contradictory ;) I think @churchyard is right here
=E2=80=94 it would be pointless if they can't talk to our Bodhi server. U=
ntil
we know if they actually work with the new client, we can't judge the
impact.

Yeah, I also agree with Miro's assessment.

The interfaces to the client haven't changed with 4, but the data that
is returned from the server, and thus serialized by and expected by the
client, is what changed. Since the interfaces haven't changed, I would
expect the dependent packages to continue working unless they are using
the data fields that were altered on the server end.

On Wed, 2019-05-29 at 21:13 +0000, Shawn Heisey wrote:

Updateinfo file is not valid XML: <open file="" '="" var="" cache="" yum="" x86_64="" 7="" epel="" 92f2e15cad66d79ea1ad327e2af7af89d98e4d1="" 53d7a3e27ff41946f476af5b4-updateinfo.xml.zck',="" mode="" 'rt'="" at="" 0x7f29f8719660="">

This is a separate issue from what we are discussing here, but the good
news is that the code that caused it is now fixed and the next EPEL
compose (which I think starts in about 2.5 hours) should resolve it
(so, expect the issue to go away over the next day or three, depending
on how long it takes to propagate to the mirror you use).

That issue is documented here: https://pagure.io/releng/issue/8392

Until we know if they actually work with the new client, we can't judge the impact.

Note that the problem already happened. Updating bodhi in stable Fedoras won't make it any worse, it can only help mitigate it.

The impact here would only be on 3rd party Bodhi servers. And while we should not break that assumption of stability, being able to talk to our own Bodhi server beats that IMHO.

On Wed, 2019-05-29 at 21:13 +0000, Shawn Heisey wrote:
That issue is documented here: https://pagure.io/releng/issue/8392

I thought I was on that issue, didn't even realize I had switched to another one! Sorry for the mixup.

I'm +1 as well, for the record.

Tagged with meeting to get fast resolution.

I won't be able to attend the meeting tomorrow, but you know my vote.

Metadata Update from @churchyard:
- Issue tagged with: meeting

3 months ago

This was discussed during the FESCo meeting day:
The proposal in https://pagure.io/fesco/issue/2137#comment-573003 is APPROVED (+5, 0, 0).

Metadata Update from @zbyszek:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

3 months ago

I have submitted a package review for the bodhi3 package: https://bugzilla.redhat.com/show_bug.cgi?id=1715936

Once we have that in updates-testing, I can add Bodhi 4 to Fedora 29/30.

Here is an update for Fedora 30 that updates to Bodhi 4.0.2 and introduces a bodhi3 compat client package: https://bodhi.fedoraproject.org/updates/FEDORA-2019-352078fd2b

Metadata Update from @zbyszek:
- Issue untagged with: meeting

2 months ago

Here is an update for Fedora 29 that updates to Bodhi 4.0.2 and introduces a bodhi3 compat client package: https://bodhi.fedoraproject.org/updates/FEDORA-2019-5ce9adadc1

Login to comment on this ticket.

Metadata