#87 The "broken dependency" email does not respect ExclusiveArch hack
Opened 7 years ago by praiskup. Modified 5 years ago

In [1] devel discussion I was told that I should use ExclusiveArch tag to void putting some packages into repostiories where dependencies are not available. Even though I've done so, e.g. for vim-syntastic-d (at least I believe I've done this correctly, you can check that vim-syntastic-d is in x86_64 but not in aarch64), I get this email every day (and it is pretty annoying):

...
On aarch64:
    vim-syntastic-d-3.7.0-8.fc26.noarch requires ldc
...

[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BFW6B2JX4RYUYVYL5LAFL34KVA2DXR47/#772BEFKNTFXKISHZYZECS4ORC4OHN44H


This is the wrong place to file the issue. the ExclusiveArch for noarch is not a hack. The problem will be a packaging bug.

using rpmspec --parse to expand out the macros shows that you do not have ExclusiveArch set.

the use of macros is overly complicated in your spec file which makes things difficult

[dennis@anubis vim-syntastic (master)]$ rpmspec --parse vim-syntastic.spec |grep Exclusive
- Add ExclusiveArch tags appropriately, per discussion:
[dennis@anubis vim-syntastic (master)]$

%package d
Summary: A syntax checker for d programming language
Summary(fr): Un vérificateur de syntaxe pour le langage de programmation d
Requires: vim-syntastic = 3.7.0-8.fc25
Requires: ldc
Provides: syntastic-d = 3.7.0-8.fc25
Obsoletes: syntastic-d < 3.7.0-6

%description d
Allows checking d sources files.
%description -l fr d
Permet de vérifier les fichiers sources écrit en d.

using rpmspec --parse to expand out the macros shows that you do not have ExclusiveArch set.

the use of macros is overly complicated in your spec file which makes things difficult

[dennis@anubis vim-syntastic (master)]$ rpmspec --parse vim-syntastic.spec |grep Exclusive
- Add ExclusiveArch tags appropriately, per discussion:
[dennis@anubis vim-syntastic (master)]$

%package d
Summary: A syntax checker for d programming language
Summary(fr): Un vérificateur de syntaxe pour le langage de programmation d
Requires: vim-syntastic = 3.7.0-8.fc25
Requires: ldc
Provides: syntastic-d = 3.7.0-8.fc25
Obsoletes: syntastic-d < 3.7.0-6

%description d
Allows checking d sources files.
%description -l fr d
Permet de vérifier les fichiers sources écrit en d.

using rpmspec --parse to expand out the macros shows that you do not have ExclusiveArch set.

the use of macros is overly complicated in your spec file which makes things difficult

[dennis@anubis vim-syntastic (master)]$ rpmspec --parse vim-syntastic.spec |grep Exclusive
- Add ExclusiveArch tags appropriately, per discussion:
[dennis@anubis vim-syntastic (master)]$

%package d
Summary: A syntax checker for d programming language
Summary(fr): Un vérificateur de syntaxe pour le langage de programmation d
Requires: vim-syntastic = 3.7.0-8.fc25
Requires: ldc
Provides: syntastic-d = 3.7.0-8.fc25
Obsoletes: syntastic-d < 3.7.0-6

%description d
Allows checking d sources files.
%description -l fr d
Permet de vérifier les fichiers sources écrit en d.

using rpmspec --parse to expand out the macros shows that you do not have ExclusiveArch set.

the use of macros is overly complicated in your spec file which makes things difficult

[dennis@anubis vim-syntastic (master)]$ rpmspec --parse vim-syntastic.spec |grep Exclusive
- Add ExclusiveArch tags appropriately, per discussion:
[dennis@anubis vim-syntastic (master)]$

%package d
Summary: A syntax checker for d programming language
Summary(fr): Un vérificateur de syntaxe pour le langage de programmation d
Requires: vim-syntastic = 3.7.0-8.fc25
Requires: ldc
Provides: syntastic-d = 3.7.0-8.fc25
Obsoletes: syntastic-d < 3.7.0-6

%description d
Allows checking d sources files.
%description -l fr d
Permet de vérifier les fichiers sources écrit en d.

@ausil changed the status to Invalid

7 years ago

I don't think so, if the ExclusiveArch wasn't there, the package would exist in the aarch64 repository, right?

What I see (on Fedora 24 box) is:

ExclusiveArch: i386 i486 i586 i686 pentium3 pentium4 athlon geode x86_64 sparc sparcv9 ia64 armv3l armv4b armv4l armv4tl armv5tel armv5tejl armv6l armv6hl armv7l armv7hl armv7hnl alpha s390x ppc ppc64 ppc64le noarch
ExclusiveArch: i386 i486 i586 i686 pentium3 pentium4 athlon geode x86_64 noarch
ExclusiveArch: i386 i486 i586 i686 pentium3 pentium4 athlon geode x86_64 armv3l armv4b armv4l armv4tl armv5tel armv5tejl armv6l armv6hl armv7l armv7hl armv7hnl noarch
- Add ExclusiveArch tags appropriately, per discussion:

I mean when I run rpmspec --parse vim-syntastic.spec |grep Exclusive. Yup, that can be rpmspec (rpm) issue.

@praiskup changed the status to Open

7 years ago

And yes, ExclusiveArch in this case is a hack. As a packager, I build noarch package and I don't really care what architectures are going distribute this binary package. I just want to build it once -- and once if all dependencies are available, aarch64 can take the built package.

http://dl.fedoraproject.org/pub/fedora-secondary/development/rawhide/Everything/aarch64/os/Packages/v/vim-syntastic-d-3.7.0-8.fc26.noarch.rpm

vim-syntastic-d is in the aarch64 tree. your package is technically not noarch as not all of the deps are available on all arches. the way to limit it is to add an ExcludeArch and the compose toopls will not put it in a given arch. your usage of macors makes your package had to trace down, but the end result is there is no ExclusiveArch lines in the spec file. if there was it would also show up in the compose logs being limited https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20161005.n.0/logs/global/pungi.global.log however you can see there is no mention of your package in the logs, but you can see plenty of others that are limited.

the bug is in your spec. I however can not tell you how to fix it correctly because your spec is not typical, and not a way I have ever worked with a spec before.

Thanks for having a deeper look, I'll debug that spec file and I'll submit appropriate bugs, because there seem to be some differences among fedora versions, my rpmspec really sees ExclusiveArch, for example.

I started debugging this, but I'm unable to get box where this doesn't work:

$ fedpkg clone -a vim-syntastic && cd vim-syntastic
$ rpmspec --parse *.spec | grep '%package d$' -A7
%package d
Summary:        A syntax checker for d programming language
Summary(fr):    Un vérificateur de syntaxe pour le langage de programmation d
Requires:       vim-syntastic =  3.7.0-9.el7
Requires:       ldc
Provides:       syntastic-d = 3.7.0-9.el7
Obsoletes:      syntastic-d < 3.7.0-6
ExclusiveArch: %ldc_arches noarch

FWIW, that output is from F24, where %ldc_arches is not yet available. OTOH on Fedora Rawhide, there's:

$ rpmspec --parse *.spec | grep '%package d$' -A7
%package d
Summary:        A syntax checker for d programming language
Summary(fr):    Un vérificateur de syntaxe pour le langage de programmation d
Requires:       vim-syntastic =  3.7.0-9.fc26
Requires:       ldc
Provides:       syntastic-d = 3.7.0-9.fc26
Obsoletes:      syntastic-d < 3.7.0-6
ExclusiveArch: i386 i486 i586 i686 pentium3 pentium4 athlon geode x86_64 armv3l armv4b armv4l armv4tl armv5tel armv5tejl armv6l armv6hl armv7l armv7hl armv7hnl ppc64 ppc64p7 ppc64le noarch

I'm not sure, @ausil, can you help me with reproducer?

I'm a bit lost. Sorry for the long post. Having a look at pungi code and
the log @ausil posted [1]:

 # => pungi/phases/pkgset/pkgsets.py:124
 def merge(self, other, primary_arch, arch_list):
     ....
     for arch in arch_list:
         self.rpms_by_arch.setdefault(arch, [])
         for i in other.rpms_by_arch.get(arch, []):
             ....
             if arch == "noarch":
                 if i.excludearch and set(i.excludearch) & set(exclusivearch_list):
                     self.log_debug("Excluding (EXCLUDEARCH: %s): %s"
                                    % (sorted(set(i.excludearch)), i.file_name))
                     continue
                 if i.exclusivearch and not (set(i.exclusivearch) & set(exclusivearch_list)):
                     self.log_debug("Excluding (EXCLUSIVEARCH: %s): %s"
                                    % (sorted(set(i.exclusivearch)), i.file_name))
                     continue

             if arch in ("nosrc", "src"):

It looks like pungi uses 'i.exclusivearch' flag to make the decision.
That flag comes from kobo code :) wich uses koji, ... for something, etc,
etc, but if I read kobo correctly:

# => kobo/pkgset.py:172
class SimpleRpmWrapper(FileWrapper):
    ...
    def __init__(self, file_path, **kwargs):
        ...
        self.exclusivearch = kobo.rpmlib.get_header_field(header, "exclusivearch")
    ...

Which is something to the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#! /usr/bin/python

import sys
import rpm

if len(sys.argv) < 3:
    print "two arguments required, TAG & FILENAME"
    sys.exit(1)

# have a look at kobo for tread safety before you'll use this!
ts = rpm.TransactionSet()
ts.setKeyring(rpm.keyring())
ts.setVSFlags(rpm._RPMVSF_NOSIGNATURES|rpm._RPMVSF_NODIGESTS)

tag = "RPMTAG_{}".format(sys.argv[1].upper())
hdr_key = getattr(rpm, tag, None)

with open(sys.argv[2], "r") as fo:
    header = ts.hdrFromFdno(fo.fileno())
    print tag + ": " + str(header[hdr_key])

Run it like ./script exclusivearch RPM:

$ ./script exclusivearch ./syslinux-nonlinux-6.04-0.1.fc25.noarch.rpm
RPMTAG_EXCLUSIVEARCH: []
$ ./script arch ./syslinux-nonlinux-6.04-0.1.fc25.noarch.rpm
RPMTAG_ARCH: noarch
$ ./script exclusivearch /var/lib/mock/fedora-rawhide-x86_64/result/dustmite-1-28.20150515git3498068.fc26.src.rpm
RPMTAG_EXCLUSIVEARCH: ['i386', 'i486', 'i586', 'i686', 'pentium3', 'pentium4', 'athlon', 'geode', 'x86_64', 'armv3l', 'armv4b', 'armv4l', 'armv4tl', 'armv5tel', 'armv5tejl', 'armv6l', 'armv6hl', 'armv7l', 'armv7hl', 'armv7hnl', 'ppc64', 'ppc64p7', 'ppc64le']

Asked @pmatilai, but thats probably correct -> exclusive arch tag should be available for source packages only. I probably misread pungi implementation?

[1] https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/logs/global/pungi.global.log

IOW, I'd like to hear two things:
- what's the broken rpmspec you have on your box, so I can fill a bug report
- what code is responsible for putting 'vim-syntastic-d noarch' package into aarch64 repository, and if I've picked the right code :) then please explain that algorithm a bit for the sake of this bugreport -- so I can understand how the vim-syntastic package should be changed to comply.

Thanks! So it means that we inherit ExclusiveArch from the "main" package? But I set the ExclusiveArch for the subpackage only. Well, it is weird anyway -> when I try the "test script" from comment above, it tells me:

$ ./script exclusivearch quick-package-20161114_1410-0.src.rpm
RPMTAG_EXCLUSIVEARCH: ['i386', 'noarch']

.. Even when I set ExclusiveArch on subpackage.

The example:
http://praiskup.fedorapeople.org/quick-package-20161114_1410-0.src.rpm

Yeah, and when you add another subpackage with different ExclusiveArch, your script will return all of them.

@lsedlar it implies that setting ExclusiveArch on subpackage is basically a bad thing ATM -> becuase pungi does not handle that correctly, right?

Yeah, it looks to me that way. Although I think the problem is closer to the RPM level: you can't really have two subpackages with different ExclusiveArch and there isn't a good way to fix it in pungi, as it doesn't get the full and correct information.

Ok, we noticed the first issue (don't expect it will work for subpackages). I'll file appropriate issue against RPM, probably ... I still think this still is hack and misuse of ExclusiveArch tag for "noarch" packages, so some cool solution should be invented.

OTOH, what I see with vim-syntastic-3.8.0-1.fc26.src.rpm is this:

RPMTAG_EXCLUSIVEARCH: ['i386', 'i486', 'i586', 'i686', 'pentium3', 'pentium4', 'athlon', 'geode', 'x86_64', 'sparc', 'sparcv9', 'ia64', 'armv3l', 'armv4b', 'armv4l', 'armv4tl', 'armv5tel', 'armv5tejl', 'armv6l', 'armv6hl', 'armv7l', 'armv7hl', 'armv7hnl', 'alpha', 's390x', 'ppc', 'ppc64', 'ppc64le', 'noarch', '%ldc_arches', 'noarch', 'i386', 'i486', 'i586', 'i686', 'pentium3', 'pentium4', 'athlon', 'geode', 'x86_64', 'armv3l', 'armv4b', 'armv4l', 'armv4tl', 'armv5tel', 'armv5tejl', 'armv6l', 'armv6hl', 'armv7l', 'armv7hl', 'armv7hnl', 'noarch']

I see at least two other issues, (a) aarch64 is not mentioned -> so why pungi puts vim-syntastic into aarch64 repository? (b) the %ldc_arches macro is not expanded, but that's not "that" important, because it contains:
i386 i486 i586 i686 pentium3 pentium4 athlon geode x86_64 armv3l armv4b armv4l armv4tl armv5tel armv5tejl armv6l armv6hl armv7l armv7hl armv7hnl ppc64 ppc64p7 ppc64le

Ah, the (b) is probably non-problem, the tag was baked into that src.rpm on Fedora 24, where is no %ldc_arches macro (and ExclusiveArch tag is probably not a subject of rpm macro expansion).

Right, I think it's the noarch bit. In the snippet above exclusivearch_list contains a list of arches compatible with aarch64, which is actually just aarch64 and noarch.

Hm, that might be the issue - because noarch is there listed intentionally -> otherwise I'm unable to build BuildArch: noarch package.

Good, that's:
https://bugzilla.redhat.com/show_bug.cgi?id=1298668

(edit: IOW it works in Rawhide, but did not work on my F24 box)

I filed the bug https://bugzilla.redhat.com/show_bug.cgi?id=1394853 , @lsedlar can you please discuss there what are the rel-eng requirements from this feature?

The last question probably, @ausil, what box have you run the broken rpmspec on?

@lsedlar, per rhbz#1394853 discussion, would it be possible to have some
git repository with list of packages that are note expected to land particular
repositories? Say package-foo-subpackage-baz: [ aarch64, armv7hl ]?

Login to comment on this ticket.

Metadata