The root of this is one of the primary issues with hiding details behind macros: if nothing documents what happens under the hood, the packaging process breaks down when the macros don't cover your particular use case. And it doesn't help that this is one of the oldest sections of the packaging guidelines, though I think it has been rewritten within the past decade. Whenever that happened, it was done without adding any explanation of what happens under the hood.

Anyway, all %find_lang does is generate a file list with translations properly marked with %lang() tags. I'm honestly not sure what those tags actually get used for, or whether it is truly essential that every locale file get marked in some way.

Our options include:

• Extending find-lang to look in more places. Note that if the translation files are included in the same tree along with the rest of the package files (as opposed to living over in /usr/share/locale or whatever) then you would then have to take care to use %exclude properly or files would probably be listed twice.

• Explicitly requiring proper %lang tags and then discussing %find_lang in the context of helpfully providing them in some situations.

• Explicitly exempting translations outside of specific directories.

• Reconsidering %find_lang entirely. Perhaps it no longer serves any useful purpose.

• Doing nothing and hoping packagers can just figure it out.

We talked about this at this weeks meeting (https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2021-03-25/fpc.2021-03-25-16.01.txt):

• #1058 How to handle %lang files in package owned directories?
  (geppetto, 16:30:42)
• ACTION: tibbs to ask if anyone knows what gettext is supposed to do
  (geppetto, 16:53:18)

So the questions I have:

• Are .mo files buried down under some private library directory like that actually useful? If so, how does gettext (or whatever library is relevant) actually find them? This informs whether we would need to try to restrict such files to being in a particular part of the directory tree.

• Is %find_lang actually supposed to find the above files? If so (and they work when located there) then why doesn't it appear to work?

Knowing the answers to those will help decide what to do next.

• This is pretty common for Python packages, using the Python gettext bindings. I saw (and created) some packages that move the .mo files into /usr/share/locale and add a symbolic link to /usr/share/locale from the in-package location (e.g. mv %{buildroot}%{python3_stileib}/thonny/locale %{buildroot}/locale && ln -s %{_datadir}/locale %{python3_stileib}/thonny/locale). That makes %find_lang work but I am unsure if this is what we want packagers to do. tl;dr they are useful, gettext finds them if upstream uses it like this.

• %find_lang is pretty much designed to find .mo files in "shared" directories, because this is the "proper way". Making %find_lang support in-package directories is tricky, because unlike the shared dirs, this time the package also needs to own the directory hierarchy (at least up to directories below %{python3_stileib} in this case).

Yes, I can verify that the python gettext module will basically look where the python code is when searching for locale files. I'm not sure it is general even necessary to make a symlink if the locale files are stored under the usual /usr/share/locale hierarchy, because I believe gettext will still find them.

Of course, the thonny package is more difficult because it keeps other items in with the locale files (specifically, there are translations of help files as well as some .bat files which probably shouldn't be packaged). So I suspect that this package really wasn't the best example, since it does something which is simply different from how most packages would do things. Even if it does expose the fact that we say "use %find_lang" instead of saying what we really want.

I believe in many instances %find_lang could be used with "in-package" locale directories simply by using the proper %exclude statement(s) and probably some basic tweaks to find-lang.sh regarding whether it outputs directory names but this does depend on exactly how RPM implements %exclude. However, given what thonny does, I suspect that a simple manual labeling of the files might be the simplest way out.

Anyway, at the moment I'm at:

• Stating in the guidelines what we really want, which is something like "locale files which are small and not required for proper functioning of the package SHOULD be tagged in the %files list with the appropriate %lang tags" and some mention of how this facilitates RPM filtering of translations. (Large locale collections should be made into langpackage anyway.) Then recommending %find_lang for this with some mention of what it does under the hood and a description of when it helps and when it doesn't.

• Investigating what it would take for %find_lang to actually handle this situation (and what you would have to do with %exclude to make that work).

BTW python-notebook has:

# Exclude i18n:
%dir %{python3_sitelib}/notebook/
%{python3_sitelib}/notebook/[_a-hj-z]*

# Language files (could be scripted, but is short)
%dir %{python3_sitelib}/notebook/i18n/
%{python3_sitelib}/notebook/i18n/*.py
%{python3_sitelib}/notebook/i18n/__pycache__/
%lang(fr) %{python3_sitelib}/notebook/i18n/fr_FR/
%lang(ja) %{python3_sitelib}/notebook/i18n/ja_JP/
%lang(nl) %{python3_sitelib}/notebook/i18n/nl/
%lang(ru) %{python3_sitelib}/notebook/i18n/ru_RU/
%lang(zh) %{python3_sitelib}/notebook/i18n/zh_CN/

It works, but it is tedious.

• Investigating what it would take for %find_lang to actually handle this situation (and what you would have to do with %exclude to make that work).

Letting it find translations that aren’t in a directory called locale would go a long way.

A workaround I suggested in a package review before finding this issue and seeing any other approaches was:

# Work around neither %%pyproject_save_files nor %%find_lang supporting
# language files that are not in a directory named “locale”:
sed -r -i '/\/translations\/.*\.mo/d' '%{pyproject_files}'
cp -rp '%{buildroot}/%{python3_sitelib}/flask_security/translations' \
    '%{buildroot}/%{python3_sitelib}/flask_security/locale'
find '%{buildroot}/%{python3_sitelib}/flask_security/locale' \
    -type f ! -name '*.mo' -delete
%find_lang flask_security
rm -rf '%{buildroot}/%{python3_sitelib}/flask_security/locale'
sed -r -i 's@/locale/@/translations/@' flask_security.lang

…

%files -n python3-%{pkg_name} -f %{pyproject_files} -f flask_security.lang

Here, I’m just filtering the .mo files out of the pyproject-rpm-macros files list, copying the directory tree with .mo files to a directory named locale, applying %find_lang, and then fixing the paths and removing the locale directory.

Similarly, this works but is tedious. It does have the advantage that I can get %find_lang to generate the tags, and don't have to re-invent that particular wheel.

If you need to tell %pyproject_save_files where to find locales, we can extend the API. Currently, the code hardcodes "locale", but we can make it configurable: https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/pyproject_save_files.py#_135

Can this issue be closed, or was there something else that you wanted the FPC to discuss/decide? I'm moving the issue to needinfo because nothing has really happened in months, but it might be better to just close and reference it in a new issue if you have follow on questions/requests.

If you need to tell %pyproject_save_files where to find locales, we can extend the API. Currently, the code hardcodes "locale", but we can make it configurable: https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/pyproject_save_files.py#_135

FTR https://src.fedoraproject.org/rpms/pyproject-rpm-macros/c/7ba21ea4a8ea88767d26be4852d8d8a6ccc1d4c2?branch=rawhide