#47 resolve-deps: Add a --json flag to get full output in JSON form
Closed by ncoghlan. Opened by otaylor.
modularity/ otaylor/fedmod json-output  into  master

Download 47.patch

When --json is passed to fedmod resolve-deps, return output in JSON form that contains, for each package

  • The NVR of the package
  • The NVR of the corresponding source package
  • The package in the transaction that satisfied each requirement

This is intended to allow other tools to provide higher-level reporting or analysis while sharing the underlying data sources and dependency-reporting logic of fedmod.

In particular, I'm using this for tooling to create module contents based based on the contents of the "upstream" flatpak runtimes. Example report

I haven't updated tests or docs in this PR yet - wanted to get feedback before I went ahead and did that.

The original attempt at this produced output like:

{
    "requires": {
        "libc.so.6(GLIBC_2.14)(64bit)": [
            "glibc-2.26-15.fc27.x86_64",
         ], 
        "rtld(GNU_HASH)": [
             "glibc-2.26-15.fc27.x86_64"
         ]
    },
    "rpm": "libtasn1-4.12-3.fc27.x86_64",
    "srpm": "libtasn1-4.12-3.fc27.src"
},

Where every dependency is a single element list. The second patch in this PR reduces accuracy for simplicity and just picks one in the rare case when multiple packages in the transaction provide a dependency. I think this is an OK tradeoff, but the lists could be left if desired.

Thanks for working on this!

After some of our earlier experiences with the "just pick one" shortcut for choosing which module to rely on for a particular RPM, I think we're going to be better off keeping the lists in the fedmod layer's output.

My rationale for that is that it's relatively easy for a wrapper layer to turn single-item lists into plain strings, but if we actually throw any raw data away, then there's no way for a wrapper layer to recover it.

The core approach you've taken in the PR looks fine to me, so my main comment is just what you already mentioned: it needs doc and test updates to match the revised implementation.

Thinking further about the list-or-not question, given that JSON is a typed format, we could potentially collapse single-element lists as a string, while leaving multi-element lists as lists. That would avoid ever losing data, while still simplifying the output for the typical case (i.e. no ambiguous dependencies).

I don't like the idea of flexible typing. Since the list of what package satisfies a requirement is 99.9% of the time a single element, having it only be a list when it's a multiple element is an invitation for people to write code that falls over in the 0.1% case.

To make the question a bit less nebulous: the only example I could find/think up: fedmod resolve-deps --json fontconfig dejavu-sans-mono-fonts liberation-sans-fonts --module=platform. fontconfig has an 'font(:lang=en)' requirement that is satisfied by both dejavu-sans-mono-fonts and liberation-sans-fonts.

So you end up with:

   "requires": {
        "/sbin/ldconfig": [
            "glibc-2.26-15.fc27.x86_64"
        ],
        "font(:lang=en)": [
            "dejavu-sans-mono-fonts-2.35-5.fc27.noarch",
            "liberation-sans-fonts-1:1.07.4-9.fc27.noarch"
        ],
        "fontpackages-filesystem": [
            "fontpackages-filesystem-1.44-19.fc27.noarch"
        ],
        "libc.so.6(GLIBC_2.14)(64bit)": [
            "glibc-2.26-15.fc27.x86_64"
        ],
        "libexpat.so.1()(64bit)": [
            "expat-2.2.4-1.fc27.x86_64"
        ],
        "libfreetype.so.6()(64bit)": [
            "freetype-2.8-6.fc27.x86_64"
        ],
        "libpthread.so.0()(64bit)": [
            "glibc-2.26-15.fc27.x86_64"
        ],
        "libpthread.so.0(GLIBC_2.2.5)(64bit)": [
            "glibc-2.26-15.fc27.x86_64"
        ],
        "rtld(GNU_HASH)": [
            "glibc-2.26-15.fc27.x86_64"
        ]
    },

My use case is "what pulled the package into the transaction" and here were both specified directly - that's why there is the duplication. But maybe the extra information is useful to someone... I was just trying to keep things a bit simpler.

Anyways, I don't mind leaving it in list form - but I don't want to make it an either/or because that seems non-robust to me.

OK, let's go with the list form then - it's easy enough to flatten for display purposes, and it makes it more likely consumers will be written to handle the "multiple providers" case.

I just migrated the CLI option handling fully over to click, so this will need to move up to be a new @click.option entry on the def resolve_deps command.

The decorator call will need to be something like:

@click.option("--json", is_flag=True, default=False, help="Output dependencies in JSON format with extra information.")

and then add a new json parameter to the function.

With the migrated CLI handling, this will be json_output=json in the resolve_deps function.

@otaylor Would you mind if I took this PR over, updated it for the revised CLI infrastructure, and switched it to always reporting lists for the "usually-only-one-but-sometimes-more-than-one" fields?

I've created an updated PR at https://pagure.io/modularity/fedmod/pull-request/63 with the above changes.

I'll add tests & docs and merge it tomorrow.

Pull-Request has been closed by ncoghlan

Metadata