What I find problematic about the screenshots is that they are not stored on the same download servers (and mirrors) as the RPM packages. It sounds wrong to me to have an unknown number of users download screenshots directly from upstream web pages when using the GNOME Software tool.

Replying to [comment:1 mschwendt]:

What I find problematic about the screenshots is that they are not stored on the same download servers (and mirrors) as the RPM packages.

Partially correct, the upstream AppData source of these screenshots is indeed upstream (as we want them to be) but during repo-compose we download these screenshots (which are of course cached from day-to-day) and rewrite the URL in the AppStream metadata to point to the distro mirror. The files are then manually uploaded onto the Fedora mirror network if there are any changes or additions.

It sounds wrong to me to have an unknown number of users download screenshots directly from upstream web pages when using the GNOME Software tool.

I agree, both from a "upstream does not have the bandwidth" point and also from the privacy point of view. This is why http://alt.fedoraproject.org/pub/alt/screenshots/ exists :)

Ok, so the FPC wanted to merge the wording for this and desktop files so better reflect the intent.

The current wording for desktop files is:

If a package contains a GUI application, then it needs to also include a properly installed .desktop file. For the purposes of these guidelines, a GUI application is defined as any application which draws an X window and runs from within that window.

Proposed new wording would be:

If a package contains a GUI application, that could be found and started directly by the user then it MUST also include a properly installed .desktop file. For the purposes of these guidelines, a GUI application is defined as any application which draws an X window and runs from within that window.

...for the AppData policy the intent would be the same "if you have an Application that the user could find/install/run/whatever then it MUST ... "

From a packager's point of view, I had many questions which weren't answered on http://people.freedesktop.org/~hughsient/appdata/. Thanks to many friendly people on IRC, I got most of them answered. However, IMHO it is necessary to have a Fedora-specific reference/documentation which can be used during package reviews or for helping the packagers creating the appdata files.

Appdata workflow:
- There is no summary which explains how the dataflow for the appdata is realized. It would be good if a complete overview would be available as Feature page in the Fedora wiki.
- As far as I understood, it works currently like this:
- packagers put <packagename>.appdata.xml files into /usr/share/packages in the respective packages
- installed <packagename>.appdata.xml files are not used at run-time
- external script runs (every x hours/days/weeks), gathers the appdata files from all packages, downloads the screenshots and gathers additional rpm metadata and compiles a global appdata xml file which contains the information for all packages (in F20: /usr/share/app-info/xmls/fedora-20.xml.gz)
- the script will also copy all screenshots from the upstream servers to the fedora servers
- the global appdata xml is distributed every x hours/days/weeks (in F20 via gnome-software package)
- Open questions:
- On which fedora servers will the screenshots be copied to? How often will they be updated? Will they be mirrored to the package mirrors?
- How long does it take to make changes to the appdata file of a package visible to the end users?
- Will the distribution of the global appdata file be changed or will it always be distributed via the gnome-software package?

Packaging Guidelines
- There is no section in the packaging guidelines with respect to appdata.
- Any licensing issues since we add additional files to the packages which have a different license than the rest of the package? Does that license needs to be mentioned in the spec file?
- Add a link to http://blogs.gnome.org/hughsie/2013/10/08/how-to-take-169-screenshots/ in order to help the packagers to create the necessary screenshots very easily (btw: thanks for that extension, it really helped!).

Testing
- How can packagers test how the content of their appdata file will be eventually presented?
(Just checking for consistency via appdata-validate is not sufficient and waiting until a new global appdata file is compiled and distributed takes a rather long time.)
- How will the translation be handled? Solely upstream?
- Which kind of software will make use of appdata? Only gnome-software or are there any other package/software tools which can display it?

Replying to [comment:4 chkr]:

From a packager's point of view, I had many questions which weren't answered on http://people.freedesktop.org/~hughsient/appdata/.

I'm sure we can update that page with some of this info, the other distros are basically going the same thing with slightly different tools.

Appdata workflow:
- packagers put <packagename>.appdata.xml files into /usr/share/packages in the respective packages

/usr/share/appdata -- this is important :)

• installed <packagename>.appdata.xml files are not used at run-time

It's <application-id>.appdata.xml not the packagename, although for most packages it's the same. And new versions of gnome-software will use the AppData installed locally and not in the distro metadata to allow you to preview the file you've created (without screenshots).

• On which fedora servers will the screenshots be copied to? How often will they be updated? Will they be mirrored to the package mirrors?

I'm scp'ing to http://alt.fedoraproject.org/pub/alt/screenshots/ which does the mirroring automatically AIUI.

• How long does it take to make changes to the appdata file of a package visible to the end users?

At the moment, the appstream file is shipped in gnome-software until Fedora servers can generate the metadata. I usually update this every couple of months on a stable release.

• Will the distribution of the global appdata file be changed or will it always be distributed via the gnome-software package?

Long term goal is to get this shipped with the other metadata; a few hurdles remain.

• Any licensing issues since we add additional files to the packages which have a different license than the rest of the package? Does that license needs to be mentioned in the spec file?

I think so, yes.

Testing
- How can packagers test how the content of their appdata file will be eventually presented?

See above, modulo screenshots as there's a very tiny privacy risk if we enable those too.

• How will the translation be handled? Solely upstream?

Yes.

• Which kind of software will make use of appdata? Only gnome-software or are there any other package/software tools which can display it?

There's Apper for KDE and a new beta-quality app-search engine that uses the data too.

I don't think forcing all GUI apps to comply with this GNOME-specific "standard" (which has little to no upstream uptake outside of GNOME, as per your own numbers) makes any sense. Unlike .desktop files, which are provided by most upstream projects, you say yourself that 89% of all GUI apps do '''not''' provide !AppData. Therefore, your proposal would make it the burden of the packager to do so. You also implicitly expect packagers to provide screenshots, which is even more unrealistic than the XML file itself.

To elaborate on the "GNOME-specific" part: Several GNU/Linux distributions and desktop package management tools (e.g. Apper) have decided to standardize on a metadata format called !AppStream that gives the complete metadata for the applications available in a repository. But what has unfortunately not been agreed on is '''how''' to generate the !AppStream file. This is where !AppData comes into play, basically another XML format that contains the data for one application, that can then be collected into !AppStream. That format has been proposed unilaterally by GNOME, and upstream uptake outside of GNOME is minimal. It is, however, the only method implemented in Fedora, and also the only method that has '''any''' upstream support. (Other distros are maintaining their !AppStream data entirely in the distro, which would only make the problem worse. So I am not proposing that we do that.)

I would also like to point out that, as long as the !AppStream file is shipped in the gnome-software package, it is essentially useless for Apper and any other application that wants to use it. This is '''not''' an appropriate location for the !AppStream file. We need this fixed by Fedora 21 if we are to ship Apper with !AppStream support enabled. (It is currently enabled in Rawhide, but it will probably get turned off for the release if this issue does not get sorted out.)

This proposal is even worse than Debian's manpage requirement, which was also proposed in Fedora and rejected by FPC due to the unjust burden it puts on the packager. Therefore, I strongly urge FPC to also reject this proposal. This is an upstream issue that cannot be solved by a Fedora policy, it is not fair to expect the packager to do this work.

Replying to [comment:6 kkofler]:

I don't think forcing all GUI apps to comply with this GNOME-specific "standard"

Kevin, can we get back to reality please? KDE projects are adding AppData files every day; there are several people in the KDE team actively supporting this initiative. Just talk to Matthias Klumpp or do a quick search of KDE svn/git and you'll see the files popping up. There's nothing GNOME specific about this at all, and the only noteworthy thing about the spec and GNOME is that there was a successful GNOME Goal a few months ago trying to support the specification better, hence GNOME is far in the lead.

that 89% of all GUI apps do '''not''' provide !AppData.

It's down to 78% now in rawhide. It's still a large amount, but there are a lot of shitty unmaintained/abandoned applications in Fedora right now. Most of the applications we care about in the desktop team already have AppData.

You also implicitly expect packagers to provide screenshots

Surely the packager is capable of running the application and sending some screenshots upstream? Usually it's just a case of finding existing upstream screenshots, and referencing them in the file.

have decided to standardize on a metadata format called !AppStream that gives the complete metadata for the applications available in a repository

Correct.

But what has unfortunately not been agreed on is '''how''' to generate the !AppStream file.

Yes it has. There's a GSoC project running now for Debian, Suse generates it during repocompose and Fedora is using createrepo_as. We're using different tools to do the same thing, but this is a requirement of the distro policy, e.g. Debian has to use yaml as a interchange format between mirrors.

That format has been proposed unilaterally by GNOME

No, it's been proposed by me, please stop spreading FUD.

and upstream uptake outside of GNOME is minimal

Btzz.

Other distros are maintaining their !AppStream data entirely in the distro

Ubuntu is using app-info (based on desktop files), rather than AppStream.

This is '''not''' an appropriate location for the !AppStream file.

Agreed, and I'm working with Dennis to get the generation done on the compose server like Suse does. I'm 90% there, but we're still tweaking the extractor. I'd be happy to push this to a separate package if we don't get all the bits worked out for f21.

but it will probably get turned off for the release if this issue does not get sorted out.)

You can do whatever you like.

This is an upstream issue that cannot be solved by a Fedora policy, it is not fair to expect the packager to do this work.

We already expect the packager to do the work by supplying missing .desktop files (or fixing up the broken ones) so they look good in the menus, so supplying another file so the app looks good in the software center isn't that much of a quantum leap. You'd hope the packager actually knows how to run the application, and is in contact with upstream. AppData files really belong upstream, and when they are missing it's usually due to an unwilling or dead upstream, so the packager has to do the work.

I've really done a lot of work reaching out to upstreams, and it's really giving dividends now. Just see https://blogs.gnome.org/hughsie/2014/05/28/appdata-progress-and-the-email-deluge/ for what I did, and you'll have to trust me that we went from 19% to 21% of apps in Fedora with AppData in under one week after those emails. I'm fully expecting that in 6 months time most of those bugzilla tickets will have paid off as the new upstream releases trickle in.

We discussed the intentions of the .desktop guidelines and this proposal in last week's meeting and came up with a few decisions.

1. We decided that the .desktop guidelines are intended to be mandatory for all GUI apps: Straw poll: should we modify the .desktop guidelines to make adding a .desktop file for GUI apps optional (+1:0, 0:0, -1:5)

2. We voted on just how strict we'd be willing to make the requirement that GUI apps include AppData.

Making it mandatory, at this time, failed. However, the votes seemed like they could change in a release or two when AppData would be more of an accepted standard (like .desktop files are now) Proposal: GUI applications (the same ones that have .desktop files) MUST also ship an AppData file so that their packages show up in application-oriented software installers (Software Center, Apper, etc).
FAILED (+1:1, 0:2, -1:1)

Making it recommended passed: Proposal: GUI applications (the same ones that have .desktop files) SHOULD also ship an AppData file so that their packages show up in application-oriented software installers (Software Center, Apper, etc). (+1:6, 0:0, -1:0)

We also noted that AppData can have different licensing terms than the software itself. We wanted the draft for AppData guidelines to explicitly mention that the metadata license must match the upstream license if we're the ones creating the AppData file. If the licenses for the AppData differ from the code license (because upstream is shipping an AppData with different licensing terms), we'll need to note the additional license in the spec file's License: field.

Please writeup a draft on how AppData is to be added that fits within these parameters (similar to the .desktop file section https://fedoraproject.org/wiki/Packaging:Guidelines#Desktop_files ) for us to vote on and we'll get it into the Guidelines.

Replying to [comment:8 toshio]:

we'll need to note the additional license in the spec file's License: field.

AppData files can only have a small subset of approved content licenses, I've added a note about that on the wiki page.

Please writeup a draft on how AppData is to be added that fits within these parameters

Something like this? https://fedoraproject.org/wiki/User:Rhughes/DraftAppDataGuidelines -- comments very welcome -- thanks.

Is there a technical reason that you are forcing projects into licensing the appdata files under a "content" license, as opposed to permitting them to have it exist under the same license terms as the rest of the project? This is the sort of ... thing that was a major roadblock for SPDX.

Replying to [comment:10 spot]:

Is there a technical reason that you are forcing projects into licensing the appdata files under a "content" license

Well, I think it's good practice to not license XML files as if they were code, but the bigger reason is so that the metadata generated from these files does not have some crazy licensing -- e.g. 'CC0 and CC-BY and CC-BY-SA and GFDL' rather than some supeset of all the package licenses in Fedora.

It's one thing to recommend and another to force. I'm generally uncomfortable with requiring that we force packagers to add a new license to existing packages, especially because this limits the likelyhood that it will be taken upstream.

FPC discussed this today and found that the example raised problems with the appdata-validate tool from f20:

{{{
$ appdata-validate --nonet comical.appdata.xml (18:02:34)
comical.appdata.xml 2 problems detected:
• markup-invalid : <id> does not have correct extension for kind
• style-invalid :

is too short
}}}

Those should be fixed so packagers have a good example.

We also noticed the utf-8 output of appdata-validate (even if locale was changed to C) which we weren't altogether happy about but is not a blocker.

The subset of licenses was seen as a blocker, though. It limits upstream being willing to take our changes, will force us into occasional conflict with upstream when we want them to then include license information for both the metadata and the code in their tarballs, and both causes extra work for packagers to update and obfuscates the value of the license tag for end users. The question also arose about what packagers would need to do if validation of an upstream AppData file failed due to invalid license. It seems that packagers would have to remove the AppData file from upstream and create a new one from scratch in order to license the metadata under a valid license. This seems counter to the desire expressed in the draft to push AppData files to upstreams.

The restrictions on metadata license will need to be resolved before this can proceed.

Replying to [comment:13 toshio]:

FPC discussed this today and found that the example raised problems with the appdata-validate tool from f20:
Those should be fixed so packagers have a good example.

Okay, I can do that today.

We also noticed the utf-8 output of appdata-validate (even if locale was changed to C) which we weren't altogether happy about but is not a blocker.

Any ideas on how to do this? I'm using the standard gettext and setlocale() calls.

The subset of licenses was seen as a blocker, though.
The restrictions on metadata license will need to be resolved before this can proceed.

Then I'm between a rock and a hard place. I wanted to do AppStream metadata a couple of years ago, and it was blocked by legal due to the crazy long license tag in the metadata package. Would an acceptable compromise be to allow a larger subset of the SPDX licences, for instance allowing Apache, BSD and GPLv3+ as well as the existing content licenses?

Richard

I would not worry about the "crazy long license tag", that issue is resolved to Red Hat's satisfaction, even if every single Free license ends up in play there. I personally stand firm on the need to permit the license of the AppData file(s) to match the license of the upstream license, which effectively means "any license permitted by Fedora".

Replying to [comment:15 spot]:

that issue is resolved to Red Hat's satisfaction

Ohh, that might have been good to know about 6 months ago...

, I personally stand firm on the need to permit the license of the AppData file(s) to match the license of the upstream license

Well, we're restricted to SPDX license IDs from the AppStream specification (as we're sharing the specification with Debian and SUSE, both who don't use Fedora terminology) , but as a compromise how about if I recommend that people choose a free content license, but allow any SPDX ID in the validation tool? I can update the AppData upstream too if this compromise is acceptable.

Richard.

I informed Matthias and Christian about the legal issue being resolved in November. I assumed that they would pass that along to you. Apologies.

I'd rather require that Fedora contributors generating new appdata for inclusion in Fedora:

• License the appdata works under the same license as the rest of the upstream codebase
• Send those changes for upstream inclusion

Obviously, if upstream provides us appdata files under their own choice of license, that is at their discretion, and assuming their license choice is Fedora-acceptable, we'll include it.

SPDX is planning on adding IDs for all Fedora Free licenses, so this should not be an issue. (I'm not sure SPDX was the right choice there, but that's your choice, and we'll work around it.)

Replying to [comment:17 spot]:

assuming their license choice is Fedora-acceptable, we'll include it.

I've changed libappstream-glib in git master to correctly validate the AppData file in this case now.

Replying to [comment:14 rhughes]:

Replying to [comment:13 toshio]:

We also noticed the utf-8 output of appdata-validate (even if locale was changed to C) which we weren't altogether happy about but is not a blocker.

Any ideas on how to do this? I'm using the standard gettext and setlocale() calls.

Use asterisks in place of the unicode bullet (or use asterisks only when the locale does not contain the unicode bullet). However, like I said, this isn't a blocker. setlocale will handle it by replacing the bullet char with a "?" in locales that don't have the unicode bullet.

@spot: We're wondering about the timeline of having Fedora approved licenses added to SPDX, though. We don't want to get into a situation where we have to omit AppData files because the SPDX data doesn't contain the licenses we need. (and we can't relicense the AppData files because we are not upstream).

@rhughes: We also wondered if we could get an example of a translation added to the example appdata.xml for summary and description. Since Fedora may care about translations more than upstream in some cases it is desirable that we can easily find the information about how to add translations to the file.

Given the likelyhood of disconnect between SPDX and Fedora licenses, we'd like to request an option be added for the appdata-validate tool that performs all checks except license validity, so that we can run that check at build time without the risk of failing on a license mismatch. (+1:5, 0:0, -1:1)

commit 6ac88cc7a9e89d3fd5a1c0602b7b4ad7ff5e239c
Author: Richard Hughes richard@hughsie.com
Date: Fri Jul 11 12:21:07 2014 +0100

Do not parse the license strings when in relaxed validation mode

I'll get a new release out to koji later today.

New release pushed into f21 and rawhide with desired validate-relax changes.

revised draft approved (+1:7, 0:0, -1:0)

Is the approved draft integrated into the packaging guidelines? I'm not seeing it. But I'm not quite sure how this particular part of the sausage-making happens. Is it waiting for someone to edit the page? Or is it somewhere official that I've just missed? Thanks!

Replying to [comment:26 mattdm]:

Is the approved draft integrated into the packaging guidelines?

I don't have permissions to edit the page; can someone with the required super-powers import https://fedoraproject.org/wiki/User:Rhughes/DraftAppDataGuidelines to just under https://fedoraproject.org/wiki/Packaging:Guidelines#Desktop_files in the wiki please? Thanks.

Hi, I have two remarks to this guidelines:

1. There are not mentioned the application addons [1], which are probably more recent then the draft
2. Since you suggest to {{{BR: libappstream-glib}}} and run some checks, it would be nice to provide some nice convenient RPM macro, which would replace the {{{%{_datadir}/appdata/}}}

[1] http://blogs.gnome.org/hughsie/2014/06/11/application-addons-in-gnome-software/

Replying to [comment:28 vondruch]:

1. There are not mentioned the application addons [1], which are probably more recent then the draft

Right; there are also other things to add too. I wanted to get at least some of the core parts of this into the package guidelines before Fedora 21. I'm quite happy to keep this ticket open and discuss future additions.

1. Since you suggest to {{{BR: libappstream-glib}}} and run some checks, it would be nice to provide some nice convenient RPM macro, which would replace the {{{%{_datadir}/appdata/}}}

Is this required? I mean, the macro name would be longer than %{_datadir}/appdata/ surely, and appdata can't change location now as it's being used in hundreds of upstream projects.

Richard

Hi, sorry if I am missing something, but I wonder about a few things ...

1) Why not to take the information from the .desktop file by default and use appdata.xml only if the author/packager wants to provide additional information that cannot be in the .desktop file?

2) What is it good for to install appdata.xml into %{_datadir}/appdata/ when the installer (Apper, GNOME Software) takes the information from /usr/share/app-info/xmls so the files in /usr/share/appdata/ will lay on end-users' systems completely unused?

3) Almost the same goes for icons, if appstream-data will copy icons to /usr/share/app-info/icons, why to have two copies of the same image?

oh, and 4)

from http://people.freedesktop.org/~hughsient/appdata/

What happens if I don't ship this file?

The GNOME Software Center currently shows a nag message that the upstream project doesn't ship the additional data. Additionally, we will penalize apps that do not ship the extra metadata by showing them lower in the search results.

why do we penalize '''users''' by hiding contents from them when some upstream just doesn't care about this stuff? (see also comment 7 about the "unjust burden")?

I'd suggest you followup onlist, one recent thread on the topic:
https://lists.fedoraproject.org/pipermail/devel/2014-January/194251.html

(trac tickets aren't exactly the best place to have a discussion)

Replying to [comment:33 rdieter]:

I'd suggest you followup onlist, one recent thread on the topic:
https://lists.fedoraproject.org/pipermail/devel/2014-January/194251.html

thanks for the pointer, but I haven't found anything that would help here - most of the thread is just an argument about inactive developers; the newer threads also do not cover these things

(trac tickets aren't exactly the best place to have a discussion)

but it is a good place to collect the answers

and, in addition, I'm not subscribed on devel as I'm not an (too much) active developer, so please forgive me if you feel this is abuse of trac ... if you could bring this up on devel and then, after the horse is beaten to death, collect the resolutions and post it here, without the discussion happening on the ticket, it'd be great

meanwhile, I have two more things:

5) If there is a trend to split localised information into separate langpacks, why to mix all locales into one file, not allowing any split?

Also, no localised screenshots allowed - "Screenshots should be taken with US English as the display language." - even if they could be tagged with language code too?

6) If we copy the screenshots, why not to provide them also in an optional package?

I've heard there are still some people who don't have Internet connection and install Fedora from DVD (do we still allow to install more packages from DVD after the initial install, right?) Or some may prefer not to get online just to browse the applications list ...

Replying to [comment:34 kvolny]:

(trac tickets aren't exactly the best place to have a discussion)
but it is a good place to collect the answers

Really, trac isn't a support forum. I'd gladly answer emails sent to me or devel@fedora, or in IRC. Listening to my talk would also probably answer a lot of those questions: https://www.youtube.com/watch?v=mSWIodEQMqo

ping FPC. Looks like this still isn't in the guidelines?

While upstream adoption has increased since I wrote comment:6, and while the !AppStream data (which is generated from the collected !AppData data) is now packaged in a way usable by all !PackageKit clients (so that particular complaint from my comment:6 is also obsolete now), I still stand by my point that this is an upstream issue, and requiring packages to provide such !AppData is an unreasonable burden on packagers.

I know that we do require .desktop files for GUI apps, but !AppData has neither the upstream adoption nor the cost-benefit ratio of a .desktop file. (Without a .desktop file, the application does not and cannot show up in menus at all. Without !AppData, the application can still be found in software managers just fine, unless they explicitly blacklist applications that don't provide it. As for the cost: If you require screenshots, providing them, in the expected resolution at that, is a lot more work than writing a simple .desktop file. But if you don't require screenshots, there will be little to no data in the !AppData that cannot already be extracted from the .desktop file.) So I think this is more like Debian's manpage requirement for every executable, which was rightfully rejected by the FPC.

Replying to [comment:38 kkofler]:

...requiring packages to provide such !AppData is an unreasonable burden on packagers.

Unreasonable? It takes ~5 minutes to write a new file from scratch and take a few screenshots. That's a lot shorter time than nearly all other parts of the package review process.

Finally written up, sorry.

Announcement text:

AppData files should be included in packages which contain graphical applications. Guidelines were added describing the use of these files: https://fedoraproject.org/wiki/Packaging:

Replying to [comment:39 rhughes]:

Replying to [comment:38 kkofler]:

...requiring packages to provide such !AppData is an unreasonable burden on packagers.

Unreasonable? It takes ~5 minutes to write a new file from scratch and take a few screenshots. That's a lot shorter time than nearly all other parts of the package review process.

I believe appdata is beyond the packaging remit and gnome should file requests directly with all upstreams if they want it.
If it only takes a short time you should have no problem doing all my packages for me as I don't have the time to waste for some gnome tool I don't give a damn about.

2014-12-31

[rave@f22-qemu ~]$ gnome-software

Segmentation fault (core dumped)

It sounds weird to add defaults for a broken package which seems to be in an alpha state.
I agree with others here that adding such a default is an unreasonable burden on packagers.
Most of us are not redhat employees as rhuges, we don't have the time and money to do that.

Replying to [comment:40 tibbs]:

Finally written up, sorry.

Great, thanks!

Replying to [comment:41 leigh123linux]:

...should file requests directly with all upstreams if they want it.

Already done: http://blogs.gnome.org/hughsie/2014/05/28/appdata-progress-and-the-email-deluge/

Replying to [comment:42 raveit65]:

[rave@f22-qemu ~]$ gnome-software
Segmentation fault (core dumped)

You need to file a report in bugzilla with a backtrace.