Hi @vashirov , thank you for reporting this issue, but I am afraid we are not able to help you here. Since the package is part of centos, please report a bug in Bugzilla. It should be under product Red Hat Enterprise Linux 8 and version CentOS Stream

I was asked by @kevin to open this ticket, so that koji devs can investigate it. Let's give them a chance to look at it.
This package is not part of CentOS, as it doesn't ship s390x packages. This package comes from RHEL8 (I took it from brew).

@tkopecek this looks like that hash mismatch we saw before?

So, was in epel, then blocked from epel and it's not properly using the external repo one's hash?

Hmm, looks like a problem in workflow. Koji already imported those rpms from EPEL. Then it was moved to RHEL and rebuilt BUT left with original NVR. This is of course conflict. New version should have had at least changed disttag if not the release. So, correct solution is to differentiate EPEL/RHEL builds - worth a bug for RHEL8?

More generally - using external repos got out of control. They are overused in many places where inheritance should have been used. Nevertheless, it is a life and I'm not sure if we should address it in koji itself. This usage breaks basic koji assumption - NVR is referring to one and only one build. We're often hitting situation when it refers to random build with same NVRs coming from different sources. Maybe we can rethink this approach (there is that old idea of "namespaces") But it will probably not solve cases like this. Maybe new thread in koji-devel?

@mikem ?

Hmm, looks like a problem in workflow. Koji already imported those rpms from EPEL. Then it was moved to RHEL and rebuilt BUT left with original NVR. This is of course conflict. New version should have had at least changed disttag if not the release. So, correct solution is to differentiate EPEL/RHEL builds - worth a bug for RHEL8?

Well, I think it's too late for that. ;(

More generally - using external repos got out of control. They are overused in many places where inheritance should have been used. Nevertheless, it is a life and I'm not sure if we should address it in koji itself. This usage breaks basic koji assumption - NVR is referring to one and only one build. We're often hitting situation when it refers to random build with same NVRs coming from different sources. Maybe we can rethink this approach (there is that old idea of "namespaces") But it will probably not solve cases like this. Maybe new thread in koji-devel?

@mikem ?

Yeah, if external repos were a different 'namespace' it might help here. Perhaps it would be helpfull also to error on this more quickly? Like when the second duplicate comes along, fail the newrepo with an error about what package is duplicate? then at least it could be detected faster?

I guess to get around this specific case I need to just adjust the db to the hash for the rhel version ?

Hmm, looks like a problem in workflow. Koji already imported those rpms from EPEL. Then it was moved to RHEL and rebuilt BUT left with original NVR. This is of course conflict. New version should have had at least changed disttag if not the release. So, correct solution is to differentiate EPEL/RHEL builds - worth a bug for RHEL8?

Well, I think it's too late for that. ;(

More generally - using external repos got out of control. They are overused in many places where inheritance should have been used. Nevertheless, it is a life and I'm not sure if we should address it in koji itself. This usage breaks basic koji assumption - NVR is referring to one and only one build. We're often hitting situation when it refers to random build with same NVRs coming from different sources. Maybe we can rethink this approach (there is that old idea of "namespaces") But it will probably not solve cases like this. Maybe new thread in koji-devel?

@mikem ?

Yeah, if external repos were a different 'namespace' it might help here. Perhaps it would be helpfull also to error on this more quickly? Like when the second duplicate comes along, fail the newrepo with an error about what package is duplicate? then at least it could be detected faster?

For external repos we are not saving everything in that. Only in the moment of the build rpms required by buildroot are installed and saved to db. So, during newRepo we don't know this. We can change this (which will make newRepo slower by importing all of the rpms in the external repo and it will clutter the db with possibly never used records). But maybe it could be the way. I've created https://pagure.io/koji/issue/2573 to look at it.

I guess to get around this specific case I need to just adjust the db to the hash for the rhel version ?
Yes, it is the easiest way.(probably all rpms from the build)

Sorry to poke, but any update on this?

So, it looks like koji is still using the 'internal' blocked one from epel:

# select * from rpminfo where payloadhash = 'be4fdd2b11bce47c00764874334295ff';                               
    id    | build_id | buildroot_id |  name   | version | release | epoch | arch  |           payloadhash          
 |  size  | buildtime  | external_repo_id | metadata_only | extra                                                  
----------+----------+--------------+---------+---------+---------+-------+-------+---------------------------------
-+--------+------------+------------------+---------------+-------                                                 
 20051676 |  1429450 |     18975271 | libzstd | 1.4.4   | 1.el8   |       | s390x | be4fdd2b11bce47c00764874334295ff
 | 248124 | 1579100758 |                0 | f             |                                                        
 21304340 |          |              | libzstd | 1.4.4   | 1.el8   |       | s390x | be4fdd2b11bce47c00764874334295ff
 | 810037 | 1579100758 |               37 | f             |                                                        
(2 rows)

So, should I delete the epel one? if I don't, I don't think it would be using the external repo one...

@kevin rpm was already used in some buildroots (koji call listBuildroots rpmID=20051676), so removing it will affect the history/rebuildability for those packages. Anyway, it could be reasonable. Would it make sense to do some rebuild of that package with just updated release and tagged to some 'override' tag?

How about we just tag the epel8 one back into epel8-override?

it's blocked in epel8, but unblocked in epel8-build so epel8-override should be still there, so I think just tagging that one in should work.

Of course then we build against the epel one not the rhel one, but practically I am not sure it matters. They are the same exact version.

@mohanboddu / @humaton what do you think? worth a try?

I think it could work, as long as epel vs rhel doesn't matter which I guess doesn't in this case as they have the same NVR.

ok. I did that. @vashirov can you try a new build now?

I just tried a build of my previously-failing nextcloud modules, it works.

@kevin, thank you, my module build was successful.

Awesome. Thanks for your patience.