@lsedlar - this was part of #778 which is something we were interested in to speed things up. Let's chat tomorrow about the limitations and the best way forward. I suspect we could just change our input json to make sure we specify all rpms by name (rather than virtual provides), but maybe there is a better way.

Thanks for everything you and @onosek do.

I'm testing it in stage and can't replicate. Using just the pungi repo or adding F27 one still works. That is using rpm-ostree-2018.1-1.fc26 (the tests were done in F26 buildroot).

I tried importing rpm-ostree-2018.4-1.fc27 and even that still works with two repos.

One possible problem could be that the newly used repo has the repodata available in the compose dir, but that repodata points to packages on the filesystem on Koji volume directly. But the builder should be able to see those as far as I can tell.

<package type="rpm">
  <name>grub2-pc</name>
  <arch>x86_64</arch>
  <version epoch="1" ver="2.02" rel="22.fc27"/>
  <location xml:base="file:///mnt/koji/" href="packages/grub2/2.02/22.fc27/data/signed/f5282ee4/x86_64/grub2-pc-2.02-22.fc27.x86_64.rpm"/>
  <format>
    <rpm:provides>
      <rpm:entry name="grub2" flags="EQ" epoch="1" ver="2.02" rel="22.fc27"/>
    </rpm:provides>
</package>

The test above is with configuration for F26. Maybe that's why the problem doesn't show.

With F27 I can reproduce with just the single pungi generated repo. Though I still don't know why it behaves the way it does.

I tried importing rpm-ostree-2018.4-1.fc27 and even that still works with two repos.

I wouldn't worry too much about ostree/rpm-ostree here. dnf has the same problems (as shown in original description) so you can just use those tools when investigating.

The test above is with configuration for F26. Maybe that's why the problem doesn't show.
With F27 I can reproduce with just the single pungi generated repo. Though I still don't know why it behaves the way it does.

Hmm, is it possible this is a bug in f27? I see the same problem with dnf from f26 though:

[root@f26vanilla ~]# rpm -q dnf
dnf-2.7.5-2.fc26.noarch
[root@f26vanilla ~]# 
[root@f26vanilla ~]# dnf install --releasever 27 --nogpgcheck --disablerepo=* --enablerepo=repo --repofrompath repo,https://kojipkgs.fedoraproject.org/compose/updates/Fedora-27-updates-20180405.0/work/x86_64/repo --installroot=/srv/ grub2 | tee
Added repo repo from https://kojipkgs.fedoraproject.org/compose/updates/Fedora-27-updates-20180405.0/work/x86_64/repo
repo                                             18 MB/s |  42 MB     00:02    
Last metadata expiration check: 0:00:09 ago on Fri 06 Apr 2018 12:40:52 PM UTC.
  * Maybe you meant: grub2
No match for argument: grub2
Error: Unable to find a match

It's surprising this shows up as a depsolve failure, I haven't debugged that but clearly:

<location xml:base="file:///mnt/koji/" href="packages/389-ds-base/1.3.7.10/1.fc27/data/signed/f5282ee4/i686/389-ds-base-1.3.7.10-1.fc27.i686.rpm"/>

Is simply not going to work outside of the Koji NFS mount environment.

Is simply not going to work outside of the Koji NFS mount environment.

Interesting. The mounts that we have mounted into the buildroot are mounts = /mnt/koji/compose/updates/Fedora-27-updates-20180405.0, /mnt/koji/compose/atomic/repo, so yes that definitely won't work in the current configuration.

Is simply not going to work outside of the Koji NFS mount environment.

It's not meant to be used anywhere else. The koji volume should be mounted on the builders by default, at least it looks fine in this task:
https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90001009
In stage there are not many packages, but still there are some.

From runroot.log:

+ mount
/dev/vda1 on / type ext4 (rw,relatime,seclabel,data=ordered)
10.5.128.139:/mnt/fedora_koji/koji on /mnt/koji type nfs4 (rw,nosuid,nodev,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.5.128.85,local_lock=none,addr=10.5.128.139)
ntap-phx2-c01-fedora01-nfs.storage.phx2.redhat.com:/fedora_koji/koji on /mnt/fedora_koji_prod/koji type nfs (ro,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.5.88.41,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.5.88.41)
...
+ ls /mnt/koji/packages
module-build-macros
perl-List-Compare
pungi
python-rpdb
rpm-ostree
testmodule
testmodule2
wmname

Another note: the repo is usable in runroot. Lorax running as part of buildinstall phase is using it since forever and works fine.

Another note: the repo is usable in runroot. Lorax running as part of buildinstall phase is using it since forever and works fine.

Yes I believe that. The problem that I was alluding to in my earlier comment is that the runroot task for ostree creation doesn't have /mnt/koji broadly mounted into it. It has /mnt/koji/compose/updates/Fedora-27-updates-20180405.0 and /mnt/koji/compose/atomic/repo. See the task from the compose on 04/05 and you can see where it has mounts = /mnt/koji/compose/updates/Fedora-27-updates-20180405.0, /mnt/koji/compose/atomic/repo, which I interpret as meaning it won't have a broad /mnt/koji/ mount in the runroot, but rather more specific ones. Maybe the more specific ones are sufficient??

I may be wrong, but I believe the mount points listed in the task are read-write. The whole of /mnt/koji should always be available read-only. However there is some problem, because the dnf commands fail even when the mount is available.

I opened a bug for DNF. This may not be really a problem there, but maybe someone more knowledgeable could help point me in the right direction.
https://bugzilla.redhat.com/show_bug.cgi?id=1565647

@lsedlar is right that /mnt/koji should also be mounted into the runroot (looking at do_mounts.log).

From IRC

10:01:00      dustymabe | lsedlar: weird.. i guess if we can't figure it out how hard would it be to revert the (move ostree to earlier phase) change?
10:01:08            <-- | mhroncok (~mhroncok@redhat/mhroncok) has quit (Quit: Leaving.)
10:01:21            <-- | Grinnz (sid183254@gateway/web/irccloud.com/x-zcprtjsepbcikeby) has quit (Quit: ~)
10:01:33            --> | Grinnz (sid183254@gateway/web/irccloud.com/x-dejnpabnfzrpngcl) has joined #fedora-releng
10:01:50        lsedlar | dustymabe, yeah, that is the fallback plan, it should be doable, but maybe a little tricky since it was a fairly large change
10:03:06            <-- | timeless (sid4015@firefox/developer/timeless) has quit (Quit: ~)
10:03:25      dustymabe | yeah. i hate that answer too :(
10:03:31      dustymabe | but only thing I can think of right now
10:03:59        lsedlar | dustymabe, I'll try to do a revert tomorrow; we can debug it with the data that we already have, so no need to block stuff on it
10:04:01            --> | timeless (sid4015@firefox/developer/timeless) has joined #fedora-releng
10:08:18            <-- | phracek (phracek@nat/redhat/x-yjkxnfbdjytnsqtn) has quit (Ping timeout: 260 seconds)
10:08:45      dustymabe | lsedlar: ok.
10:09:03      dustymabe | puiterwijk: mboddu ^^ if we can't figure out a fix then we'll revert the change that caused the ostree issues in pungi tomorrow

Summary: We will revert the change that caused this issue tomorrow if we can't figure out the problem so that we can unblock the modularity work that was done recently.

The feature is reverted in Fedora build of pungi-4.1.23-2.fc2[6789].

The problem could be caused by the fact that the repo contains source packages as well. DNF searches for grub2, finds the source package that can not be installed and does not check for other packages that provide grub2. I got a suggestion to use -x grub2.src in the example dnf call, but that still produces weird dependency errors.

--> Starting dependency resolution
--> Finished dependency resolution
Error: 
 Problem: conflicting requests
  - nothing provides which needed by grub2-pc-1:2.02-22.fc27.i686
  - nothing provides which needed by grub2-pc-1:2.02-22.fc27.x86_64

Interesting. I think that actually worked, though. You see that the error message matches the 3rd block quote from the description: nothing provides which needed by grub2-pc, which is the error I got from using the real repo. I think that it is acting right although I would want to know why there is no text after nothing provides in the error message that indicates to us what dep is missing. dnf bug?

Good point.

I tried running the whole task but telling it to exclude all source packages from the pungi repo (via exclude=*.src in repofile). That did not seem to have any effect though, nor did just excluding grub2.src.

The confusing message is actually correct: nothing provides which needed by grub2-pc-1:2.02-22.fc27.i686 says that grub2-pc depends on which and that does not exist.

hey @lsedlar - i know you were poking around on this recently. Did you have any success?

The latest progress is that this could be a regression in DNF, but so far no one from DNF team confirmed or denied that. See latest comments on https://bugzilla.redhat.com/show_bug.cgi?id=1565647

Proposed fix in libdnf: https://github.com/projectatomic/libdnf/pull/1

is this still an issue?

I tested this in stage against packages from f29 tag and the ostree was created successfully. I'll close this and remove the patches that reverted #778 from next RPM build.

thanks @lsedlar - when you do that let us know so we can make sure everything still looks good.