$ koji list-tagged epel7 libmspack
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------

Cannot submit libmspack ('0', '0.5', '0.0.6.alpha.el7') to stable since it is older than ('0', '0.5', '0.1.alpha.el7')

[11:55:59] <orionp> mboddu: FWIW - libmspack-0.5-0.1.alpha.el7 seems to be the first version pushed to epel7 - https://bodhi.fedoraproject.org/updates/?packages=libmspack
[12:14:37] <rsc> Any chance for #8228 and
[12:14:44] <rsc> #7846 (unretire/unblock)
[13:19:52] <+mboddu> orionp: If libmspack is moved to rhel, it should be blocked in epel (which it isn't), so your build should be untagged as well
[13:19:57] <+mboddu> rsc: Let me take a look
[13:27:12] <orionp> mboddu: I'm attempting to provide a limited arch package - rhel does not ship libmspack for non-x86_64
[13:30:53] <+mboddu> rsc: Done
[13:31:53] <+mboddu> orionp: Okay, now that makes sense, thanks
[13:31:58] <+mboddu> Let me see whats going on then
[13:43:02] <+mboddu> nirik: Any idea about https://pagure.io/releng/issue/8245? Its a limited arch package in rhel, so they are planing to build it for other arches, as I noted in the ticket libmspack-0.5-0.1.alpha.el7 is not in epel7 tag but bodhi is still complaining
[13:43:21] <+nirik> can look in a few
[13:43:45] <+mboddu> nirik: Thanks
[13:45:42] <+nirik> mboddu: I think that check might be in bodhi itself... not sure how to bypass it... bowlofeggs ^
[13:46:34] <+mboddu> nirik: Okay, well, learning a new thing today :)
[13:47:11] <bowlofeggs> mboddu, nirik: wouldn't it be incorrect to do this since it went out to users?
[13:48:05] <bowlofeggs> bodhi does not have a way to bypass that check, but yes it is in bodhi itself
[13:48:17] <bowlofeggs> it's one of the API validators
[13:48:17] <+mboddu> bowlofeggs: I dont have the historical knowledge of why it is designed that way, but as per docs it is the solution to fix it - https://fedoraproject.org/wiki/EPEL:Packaging#Limited_Arch_Packages
[13:48:32] <+nirik> well, not sure... rhel doesn't give us much choice when they add/remove things...
[13:48:57] <+nirik> we could ask them to push the newer one, but then it's newer than the rhel one, which is bad.
[13:49:40] <+nirik> I just wish we had a way to have better communication with this "Red Hat" company somehow.
[13:50:06] <bowlofeggs> i'm not sure i understand the connection between wanting to push an older package over a newer one and limited arches?
[13:50:17] <+nirik> this is a epel specific policy
[13:50:40] <+nirik> when rhel has a package, but it doesn't ship it on all arches, epel adds one (older than the rhel one) to do that.
[13:50:48] <+nirik> the epel one gets used in buildroots for everything.
[13:51:08] <+nirik> the rhel one hopefully is newer on the arches they ship, so we don't technically overlap packages
[13:51:35] <+nirik> if the epel one is rpm newer we are replacing rhel packages
[13:51:47] <rsc> mboddu: thanks!
[13:52:20] <bowlofeggs> so did this package switch from being in EPEL to being in RHEL, without getting a version bump (or something like that)?
[13:54:08] <+mboddu> nirik: But vice versa is also true right? if the epel rpm is older then how come epel package will be used to generate the buildroot as rhel is shipping a newer one? Or will it use rhel one for the arch specific one's and epel one for unsupported rhel arches to generate the epel buildroot?
[13:54:37] »» +mboddu needs some tea
[13:54:41] »» +mboddu bbiab
[13:55:30] <+nirik> bowlofeggs: yes, it not only didn't get bumped, it got downgraded from what was in epel
[13:55:56] <bowlofeggs> whoah
[13:56:01] <+nirik> mboddu: because koji acts on src packages. If there is a libsmbios in epel, it will use that for everything.
[13:56:18] <bowlofeggs> i definitely don't think we can use any of bodhi's features to solve this problem, and am really not sure what to do
[13:56:50] <+nirik> well, the problem is that bodhi is doing a simple version check... at least I think it's bodhi doing it
[13:56:56] <+mboddu> nirik: Ahhh, okay
[13:57:03] <bowlofeggs> yeah bodhi is making that message
[13:57:43] <bowlofeggs> nirik: https://github.com/fedora-infra/bodhi/blob/3.13.0/bodhi/server/validators.py#L1323-L1329
[13:58:34] <+mboddu> 1 stupid question/ interesting question : why didn't we hit this before? I remember doing this couple of times, is it because rhel is also shipping the latest version?
[13:58:44] <bowlofeggs> nirik: it might be possible for me to force this to happen in the db, but i don't know what the consequences of that may be
[13:59:18] <bowlofeggs> this file (validators.py) just validated REST API requests
[13:59:59] <bowlofeggs> so i can bypass it and force the update to be request stable. i just don't know what will happen to this or to future updates if we do that.
[14:00:45] <bowlofeggs> it seems like this same thing would also happen to any future updates of this package
[14:01:10] <bowlofeggs> so we can force this one through, but then we'd have to force any future updates through as well (i think)
[14:02:26] <+nirik> ideally there will be close to 0 updates for this
[14:04:13] <bowlofeggs> i'm also not sure what the composer will do if we force this through
[14:04:15] <bowlofeggs> or koji?
[14:04:32] <+nirik> all the old builds are untagged in koji
[14:04:45] <+nirik> and koji doesn't care about versions either (as long as they are unique)
[14:05:14] <+nirik> such a mess
[14:07:16] <bowlofeggs> yeah
[14:07:38] <bowlofeggs> ok, well do we want to do an FBR to try to force this through and see what happens?
[14:07:49] <bowlofeggs> i really don't feel like i can predict whether that will work
[14:08:04] <+mboddu> bowlofeggs: Actually I prefer waiting until freeze is over
[14:08:04] <+nirik> well, what are the alternatives...
[14:08:31] <bowlofeggs> nirik: i don't have any other ideas
[14:08:41] <bowlofeggs> mboddu: that's ok with me ☺
[14:08:57] <+nirik> tweek the db to let it thru, or ship the old epel one thats newer than rhel... but then we replace the rhel one... perhaps we could ask them to update it in rhel... but thats not likely going to happen quickly if at all
[14:09:33] <bowlofeggs> nirik: haven't we already shipped the older one that's newer that rhel?
[14:09:39] <bowlofeggs> i.e., hasn't that train sailed?
[14:09:53] <+mboddu> bowlofeggs: In this case or in general?
[14:10:05] <+nirik> yep. But it's somewhat different for rhel... since they have minor versions and we assume people reinstall or look at whats added/dropped at those...
[14:10:31] <bowlofeggs> mboddu: this case
[14:10:33] <+nirik> so when someone does a new install they get the rhel one, then add epel and it replaces the rhel one... which we say we won't do
[14:10:47] <bowlofeggs> nirik: oh ok
[14:11:01] <+nirik> sure all the old installs that had the epel one still do, but rhel users do fresh installs more often (IMHO)
[14:11:34] <+nirik> perhaps it's a question for epel steering
[14:12:55] <+mboddu> Can we add a flag to skip certain packages going through that validators.py check? That way we dont have to face this again.
[14:13:45] <bowlofeggs> i don't know if a flag would be good - this seems like a highly exceptional case
[14:14:05] <bowlofeggs> we wouldn't want people to do this in almost any circumstances
[14:14:21] <bowlofeggs> this is the first time i've seen this problme in 3 years, for example
[14:14:28] <bowlofeggs> so it's certainly very much an edge case
[14:15:16] <bowlofeggs> if we expect this to happen more perhaps we could make EPEL point releases in bodhi rather than just EPEL major releases?
[14:15:47] <bowlofeggs> that might reflect the expected use cases that nirik described better?
[14:16:04] <+mboddu> Then in that case, we should probably document this
[14:16:23] <+nirik> bowlofeggs: there are plans to do that yes.
[14:17:43] <bowlofeggs> mboddu, nirik: do you want to see what the EPEL SIG wants to do, or do you want me to just try to force this after the infra freeze?
[14:18:15] <+mboddu> bowlofeggs: We can ask EPEL SIG now and see what they have to say before infra freeze ends
[14:18:23] <bowlofeggs> ok
[14:18:51] <+nirik> https://fedoraproject.org/wiki/EPEL/Changes/Minor_release_based_composes
[14:19:02] <bowlofeggs> oh cool
[14:19:11] <+nirik> yeah, perhaps someone will come up with a more clever plan (but not holding my breath)
[14:19:34] <bowlofeggs> yeah i think that would just make this problem go away, because the check is whether they are in the same release or not
[14:19:59] <+nirik> well, not sure if this means different releases for bodhi or not... perhaps it does...
[14:20:09] <bowlofeggs> another "hack" i could do is to go force the other update to be "obsolete" (or some other non-stable status)
[14:20:15] <bowlofeggs> then the check will think it's ok
[14:21:01] <bowlofeggs> that might be better actually
[14:21:12] <bowlofeggs> and we could comment on the other one to say what the deal is
[14:21:20] <+nirik> yeah, and true... since we untagged it
[14:21:28] <+mboddu> bowlofeggs: Yeah, that is better
[14:21:46] <bowlofeggs> maybe "unpushed" is better than obsolete
[14:21:55] <bowlofeggs> but yeah i think that might actually work better
[14:22:14] <bowlofeggs> feels less messy anyway
[14:22:53] <+mboddu> Thanks nirik and bowlofeggs

 update to stable and got ''NoneType' object is not subscriptable'