#8159 please migrate to the new Fedora translation platform
Closed: fixed 4 months ago by abbra. Opened 10 months ago by jibecfed.

Hello, the Fedora project migrates its translation platform to Weblate [1].

This tool directly interact with your git repository, and requires us to know:

  • [mandatory] which branch is your development branch?
  • [mandatory] have you merged latest translation from Zanata and locked the project?
  • [info] Weblate will handle updates when pot file changes, don't edit po files for this [2]
  • [optional] what is the license of translation? (please use a code from https://spdx.org/licenses/)
  • [optional] do you have any announcement/warning you would like to display to the translators? (it will be displayed in Weblate)
  • [optional] do you need us to activate any specific checks? (this is a setting per component [3])
  • [optional] do you need us to automatically detect new translation files? (typical usecase: website translation with one translation file per page)

Please note:

  • For github hosted projects, Weblate open pull request. For other project, you'll have to add a key to allow commits.
  • In Weblate's vocable, one project is a group of component. Each component is a translation file. You can have many projects or many components or both.
  • You can change your mind over time, just reach trans@lists.fedoraproject.org

[1] https://communityblog.fedoraproject.org/fedora-localization-platform-migrates-to-weblate/
[2] https://docs.weblate.org/en/latest/admin/continuous.html#avoiding-merge-conflicts
[3] https://docs.weblate.org/en/latest/user/checks.html#translation-checks


Metadata Update from @abbra:
- Issue assigned to abbra

10 months ago

Thanks, @jibecfed. It'll take me some time to merge recent translations from Zanata because of a broken set of tools for communicating with Zanata in Fedora (new tools written in Python do mis-attribution of translation authorship, old tools break directory placement and do some weird changes in the metadata when retrieved/pushed).

Some answers:
- we have multiple branches developed in parallel. In Zanata we maintained several branches and periodically did merges between them. How is this going to be tracked in Weblate?
- pot file is not stored in git right now, it is build time generated. Should we store and update it in git to get weblate to notice it? Is there a way to run per-project hook to generate pot file instead?
- we host freeipa source code on pagure but work through Github pull requests workflow, so for Weblate we'd prefer a pull request to be opened on github.

For all your questions, I suggest you follow the same scheme as Anaconda: https://github.com/rhinstaller/anaconda-l10n

They have a dedicated repo for all pot and po files, with direct commit from weblates.
It makes all the thing easy to handle for everyone.
Using Pagure is fine, user to set with commit right is @weblatebot

Here are translations from zanata master to help you:

po.7z

I dont really understand your issues, please let me know if I can help you.

Last Zanata import lost translators' attribution and quite a few other things. I need now to manually undo some of those changes.
Thanks for the export, I'm working on the integration back to IPA.

@jibecfed I submitted a pull request that prepares for Weblate use: https://github.com/freeipa/freeipa/pull/4419
Once it is accepted, we can proceed with the switch.

master:

  • 92e3625 Keep ipa.pot translation file in git for weblate
  • b4722f3 Update translation infrastructure
  • 3fc932a po: update ipa.pot template
  • 0be22a6 po: update Bengali translation timestamp
  • 6cd244d po: update Catalan translation timestamp
  • 68cc049 po: update Czech translation timestamp
  • 117893f po: update German translation
  • 439c488 po: update English (United Kingdom) translation timestamp
  • 2859216 po: update Spanish translation
  • e657491 po: update Basque translation timestamp
  • 1a0232a po: update French translation
  • 35c1da8 po: update Hindi translation timestamp
  • f18a4f8 po: update Hungarian translation timestamp
  • 347d9c7 po: update Indonesian translation timestamp
  • 60d69a8 po: update Japanese translation timestamp
  • 1c30d18 po: update Kannada translation timestamp
  • 0c9066e po: update Marathi translation timestamp
  • 7f3cc11 po: update Dutch translation timestamp
  • 3e63695 po: update Punjabi translation timestamp
  • 047c8cc po: update Polish translation
  • baf1a72 po: update Portuguese translation timestamp
  • 45dede7 po: update Portuguese (Brazil) translation timestamp
  • ad3ef9d po: update Russian translation
  • ed55c40 po: update Slovak translation timestamp
  • e50c250 po: update Tajik translation timestamp
  • 9fcae15 po: update Ukrainian translation
  • 42e8669 po: update Chinese (China) translation

Failed to apply patches onto origin/ipa-4-8. Manual backport is needed.

ipa-4-8:

  • 9ff7b4a Keep ipa.pot translation file in git for weblate
  • 831f4dd Update translation infrastructure
  • e23ba77 po: update ipa.pot template
  • 16d9556 po: update Bengali translation timestamp
  • 29e3ade po: update Catalan translation timestamp
  • c8ba436 po: update Czech translation timestamp
  • 0d053d8 po: update German translation
  • 37a1e92 po: update English (United Kingdom) translation timestamp
  • 7af52df po: update Spanish translation
  • 92fb5c5 po: update Basque translation timestamp
  • a2ca393 po: update French translation
  • c4dd8b2 po: update Hindi translation timestamp
  • 595d506 po: update Hungarian translation timestamp
  • 124a563 po: update Indonesian translation timestamp
  • 89dbf88 po: update Japanese translation timestamp
  • 89b048d po: update Kannada translation timestamp
  • 28a963e po: update Marathi translation timestamp
  • e794528 po: update Dutch translation timestamp
  • e4dfb74 po: update Punjabi translation timestamp
  • 4e3867f po: update Polish translation
  • 31a9da8 po: update Portuguese translation timestamp
  • eab195f po: update Portuguese (Brazil) translation timestamp
  • db433fb po: update Russian translation
  • 3c15e47 po: update Slovak translation timestamp
  • 3d411cf po: update Tajik translation timestamp
  • 855a36b po: update Ukrainian translation
  • c6adee0 po: update Chinese (China) translation

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

7 months ago

Sorry, autoclosed for PR push but we need the other parts still to be done.

Metadata Update from @abbra:
- Issue status updated to: Open (was: Closed)

7 months ago

@jibecfed, I added weblate app to https://github.com/freeipa/freeipa and locked Zanata project.
I have no rights to create freeipa project at translate.fedoraproject.org, so I need your help.

Current state: the way weblate works doesn't really suit FreeIPA workflow. We have few requirements that aren't possible to support with Weblate:

  • .po files should have location information stripped and they should be sorted in a specific order.
  • Weblate submits .po files as they are in weblate's git repo, with Github refusing to show any content because it is too big (25-40K lines per file)
  • Weblate doesn't force-push upstream changes to own repository which makes it impossible to individually push stripped .po files as updates. It might, actually fail our way of merging pull requests because we do not merge on github and thus commit IDs will be different between what is submitted and what is committed

A proposal is to use a separate git repository for translations and let Weblate handle it. Alternatively, Weblate 4.0 might get support for VCS-less projects where PO/POT files can be uploaded separately, pretty much like it was with Zanata.

master:

  • ac5cb42 po: fix LINGUAS to use whitespace separation

ipa-4-8:

  • 616ad39 po: fix LINGUAS to use whitespace separation

master:

  • 894b3f1 po: remove zanata config since translation was moved to weblate

ipa-4-8:

  • a29eec3 po: remove zanata config since translation was moved to weblate

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 months ago

I locked your components long ago to prevent conflicts: https://translate.fedoraproject.org/changes/?project=freeipa&lang=&action=14&action=15

I totally forgot about this, I'll update your repositories and unlock them.

You may receive some useless update commits before getting actual translatiosn.

@jibecfed we still need to solve the issue with sorting and clearing up the updates. They aren't going to be accepted upstream in the current shape as provided by weblate so unlocking repositories is not helpful as it creates a false impression that the process was worked out.

oops, what are the issues?

Jean-Baptiste

ok, striping line numbers is easy=2E I'll do that=2E

commit size is depending on the size of your translation files, so if you =
want to reduce it, do multiple po files=2E It doesn't change much for trans=
lators=2E

about string order, you are the first project I see asking that, what is t=
he usecase?

about other contributions on po files they should be none=2E
Anyone you included can import po files into weblate in a few easy steps=
=2E
The issue with accepting PR prevents translators to see changes in Weblate=
and to work as a team=2E

adding a new repository wouldn't be my recommendation=2E

Both stripping and ordering are used to keep reduce the diff size for po updates. We prefer to keep a clean and lean git history.

stripping was added, let's see if this has an impact on ordering

Hey guys, I want to start a Hebrew translation, Hebrew (along with some other languages) is not handled correctly by the various terminal emulators (For example: hello will appear as olleh).

I was wondering if there's a way to ditinguish between WebUI strings and CLI strings in FreeIPA, thank you!

There is in general no way to distinguish between those two targets in the translation strings. They are not marked for a specific purpose and any client (both CLI and Web UI are just two different frontends, there could be more) can use the strings returned by the server's end-points.

Both stripping and ordering are used to keep reduce the diff size for po updates. We prefer to keep a clean and lean git history.

Stripping worked as expected: https://github.com/freeipa/freeipa/commit/963c015a0d1376145232ffe59977d04343f3c68c

rest of the commit volume is related to outdated po files:
https://github.com/freeipa/freeipa/blob/master/po/ipa.pot says: "POT-Creation-Date: 2020-06-10 22:26+0300\n"

While https://github.com/freeipa/freeipa/blob/master/po/bn_IN.po says "POT-Creation-Date: 2019-08-14 10:13+0300\n"

It makes first commits really big. There is nothing I can do against that.

After this first huge commit, you will have per release:

  • one commit updating all your po files with new strings coming from POT updates
  • one commit per po file with translations updates

The fact you can't read this pull request with Github really isn't an issue (you can read it in CLI).
You can trust Weblate for correctly doing his job. This tool is used for that for years now.
If you don't trust Weblate, do it yourself using gettext tools, the result will be the same.

If you feel like this may creates a too big git log over time, you'll be always able move the po folder from one git repository to another repo, keeping its history and remove the folder and his history from your git repository. I'll change the Weblate configuration so make that easy.

@jibecfed thanks. I can see stripping work now, that's good.

Since we are not committing to github directly, I need to check how this pull request would work with our tooling. Give me day or two to see over it, I am in the middle of something else right now.

@jibecfed I tried to look at the https://github.com/freeipa/freeipa/pull/4981.patch (weblate patch for master). It applies cleanly but after running

./autogen.sh
make -C po

in the source tree, I get a lot of differences (added, removed, file path):

$  git diff --numstat
15      23419   po/bn_IN.po
361     23990   po/ca.po
594     24906   po/cs.po
3168    24960   po/de.po
16      23191   po/en_GB.po
11436   27586   po/es.po
337     24548   po/eu.po
14455   28444   po/fr.po
30      23179   po/hi.po
87      23835   po/hu.po
147     23411   po/id.po
285     24399   po/ja.po
449     24320   po/kn.po
40      23226   po/mr.po
1       23182   po/nl.po
34      23309   po/pa.po
1038    24807   po/pl.po
140     23662   po/pt.po
334     23032   po/pt_BR.po
27280   28739   po/ru.po
1786    25371   po/sk.po
206     23768   po/tg.po
1235    22964   po/tr.po
29293   29293   po/uk.po
8536    20783   po/zh_CN.po

Effectively, the update is incompatible with our current use of GNU gettext. And this is not about stripping the file locations and reordering -- actual translations are missing. For example, looking at po/bg_IN.po, I see a removal of a translation that exists in the POT file:

$  git diff po/bn_IN.po |grep -A1 'must be an integer'
-msgid "must be an integer"
-msgstr "একটি দশমিক সংখ্যা হওয়া জরুরি"
$ cat po/ipa.pot |grep -A1 'must be an integer'
msgid "must be an integer"
msgstr ""

These changes happen over multiple translation files. Any ideas how we can get around this?

In: https://github.com/freeipa/freeipa/blob/master/po/bn_IN.po#L52

You'll see:

msgid "must be a decimal number"
msgstr "একটি দশমিক সংখ্যা হওয়া জরুরি"

In https://patch-diff.githubusercontent.com/raw/freeipa/freeipa/pull/4981.patch
I see for the same language:

+#, fuzzy
+#| msgid "must be a decimal number"
+msgid "must be an integer"
+msgstr "একটি দশমিক সংখ্যা হওয়া জরুরি"

It means Weblate detected close content and marked it as fuzzy.
The string "একটি দশমিক সংখ্যা হওয়া জরুরি" isn't the translation of "must be an integer".

The content file: https://github.com/freeipa/freeipa/blob/5b962b0ba1f7aaa00c0ba9f55efc30e1dc1e1483/po/bn_IN.po looks fine to me. You can access it through https://github.com/freeipa/freeipa/pull/4981/files and click "see file" button.

Login to comment on this ticket.

Metadata
Attachments 1
Attached 7 months ago View Comment