This is a Fedora 24 Alpha blocker.

we need to seperate the Cloud install tree from the Cloud images. the Cloud images go to /pub/fedora while the cloud install tree goes to /pub/alt

We've agreed that delivering the fix for https://pagure.io/pungi/issue/113 is enough for Alpha.

The way I understand this is that the end result is that there will be more locations with deliverables, each with their own metadata. The problem this is supposed to solve is purely how the data is delivered to users.

Does this absolutely need to happen as part of running Pungi?

It would be quite a massive change. I think it would be cleaner to implement it by changing the process instead:

• Treat output of Pungi as a build artifact. Let's call it raw compose. It is no longer a final collection of deliverables to publish.
• Have a separate tool that takes this raw compose, splits it into configured locations, and munches metadata to match what is where. These blobs of stuff would be officially published.

One advantage of this approach is that we could drop the business with empty variants – we would no longer care there are empty repos in some variants that we don't need. They would just not get moved anywhere.

I'm not completely clear on how the metadata should ultimately look (irrespective of how the splitting happens). We can't have two locations with different contents, different metadata both claiming they are the same compose. This would be ugly for importing to PDC and probably other uses.

NACK ... this is outside of pungi scope. Please use mirror manager like solution. You're solving problem on a wrong layer.

Lubos

NACK ... this is outside of pungi scope. Please use mirror manager like solution. You're solving problem on a wrong layer.
Lubos

huh? please explain yourself. we absolutely need to have things broken down and going to different namespaces. we will be shipping different arches for the same variant to different namespaces

to give an example

fedora-secondary/Server/ppc64le

fedora-secondary/Server/ppc64

fedora-secondary/Server/s390

fedora-secondary/Server/s390x

fedora-secondary/Server/i386

fedora-secondary/Workstation/aarch64

fedora-secondary/Workstation/i386

fedora/Server/aarch64

fedora/Server/armhfp

fedora/Server/x86_64

fedora/Workstation/armhfp

fedora/Workstation/x86_64

alt/Cloud/x86_64

alt/Cloud/armhfp

alt/Cloud/aarch64

alt/Cloud/s390x

alt/Cloud/ppc64le

alt/Cloud/ppc64

What we could start with is to break stuff up into different directories and keep single place with metadata.

Currently variants are directly in compose/ directory. We could push them one level deeper to create structure like this:

compose/
compose/alt/
compose/fedora/
compose/fedora-secondary/
compose/metadata/

This should be doable with just a special paths module.

What are the rules for what stuff goes into which subdirectory? Where should Spins or Labs go?

There is currently nothing for ppc or s390 arches. Is it going to be added?

Actually, splitting variants into different subdirectories will work, but splitting architectures is more challenging.

If at some point there are image tasks or livemedia tasks for both primary and secondary architectures, we no longer have a way to pass the install tree url: just substituting $basearch or $arch is not going to be enough since different arches will live in different locations.

Do we ignore such situations?

I have a simple script that should be able to carve up finished compose: see pull request 20 in compose-utils.

The usage for Fedora would be:

$ compose-partial-copy path/to/latest /mnt/fedora/alt --variant Cloud --variant Labs

$ compose-partial-copy path/to/latest /mnt/fedora/secondary --arch s390 --arch s390x --arch ppc64 --arch ppc64le --arch i386

The first one will call rsync like this rsync -avHh path/to/latest/compose/Cloud path/to/latest/compose/Labs /mnt/fedora/alt, the second one will call it multiple times (as that work can not be achieved with a single call).

The proposed script is available in compose-utils-0.1.7 which is built for all supported Fedora versions.

I have put example of how to use the utility into pungi-fedora PR 61.

I marked the issue as fixed, but the fix is not ideal solution to the problem as that would require a major change in pungi.

The problems with it are:

• The configuration for splitting is outside of regular pungi config.
• There is no metadata in the copied over content: PDC has no idea what went where, and tracking the content on mirrors to the actual compose is very difficult. To fix this we would need new metadata format, as current composeinfo can not describe a fragment of a compose.

We will need to revisit this at some point.