PR#6: update README and help messages - fedora-docs/translations-scripts

fedora-docs / translations-scripts

#6 update README and help messages

Merged 5 years ago by jibecfed. Opened 5 years ago by jibecfed.

new_readme into all-src-to-pot-python

fix navigation issue

Jibec • 5 years ago

c29d684

make po-to-src language agnostic

Jibec • 5 years ago

b5ab386

add translated-sources commit and moved commit_l10n

Jibec • 5 years ago

90da771

update README and help messages

Jean-Baptiste Holcroft • 5 years ago

a3f8ad9

README.md

file modified

+25 -52

		`@@ -1,12 +1,13 @@`
		`- # Docs Translation Prototype`
		`+ # Fedora Docs internatialization`

		`- This is a prototype of a multi-language Fedora Docs build pipeline. Work is tracked in [https://taiga.fedorainfracloud.org/project/asamalik-antora-for-docs/epic/1](https://taiga.fedorainfracloud.org/project/asamalik-antora-for-docs/epic/1).`
		`+ This is a prototype of a multi-language Fedora Docs build pipeline. Work is tracked in [https://fedoraproject.org/wiki/User:Jibecfed/fedoradoc-antora-localization](https://fedoraproject.org/wiki/User:Jibecfed/fedoradoc-antora-localization).`

		`- These scripts work with:`
		`+ The internatialization work with:`

		`- * Fedora Docs sources — sources of the docs in an adoc format.`
		`- * Translation repos — [pagure.io/projects/fedora-l10n/*](https://pagure.io/projects/fedora-l10n/%2A) — repositories holding POT and PO files, acting as a bridge between the Fedora Docs site and translators. This way, translators can choose any workflow with a translation engine of their choice. The only requirement is that they need to work with POT and PO files.`
		`- * Translated sources - [pagure.io/fedora-docs/translated-sources/](https://pagure.io/fedora-docs/translated-sources/) — all translated adoc files used to build the translated site.`
		`+ * Fedora Docs sources — sources of the docs in an adoc format. For full list of these, see [https://pagure.io/fedora-docs/docs-fp-o](site.yml in docs-fp-o).`
		`+ * Translation scripts — [https://pagure.io/fedora-docs/translations-scripts/](https://pagure.io/fedora-docs/translations-scripts/) — scripts to convert from adoc to pot and from pot to adoc.`
		`+ * Translation repos — [https://pagure.io/projects/fedora-l10n/*](https://pagure.io/projects/fedora-l10n/%2A) — repositories holding POT and PO files, acting as a bridge between the Fedora Docs site and translators. This way, translators can choose any workflow with a translation engine of their choice. The only requirement is that they need to work with POT and PO files.`
		`+ * Translated sources — [https://pagure.io/fedora-docs/translated-sources/](https://pagure.io/fedora-docs/translated-sources/) — all translated adoc files used to build the translated site.`

		`## How to run`

		`@@ -18,61 +19,33 @@`

		`### Submitting sources for translation`

		- The `all-src-to-pot.sh` holds a list of repositories to be translated.
		+ The `buid.py` script will find list of repositories to be translated in the file site.yml from the the docs-fp-o repo.
		`It clones all of the specified source repositories, all of the matching translation repos,`
		`and converts the English sources into POT files. The output of this script are local copies of the translation repos, updated with the newly generated POT files.`

		`- It takes one argument — an absolute path to an empty directory to store the output.`
		`+ It takes the following arguments :`
		`+ --clone_sources {true,false}`
		`+ Should we clone/update sources or use local content?`
		`+ --commit_l10n {true,false}`
		`+ Should we commit changes on l10n repos?`
		`+ --commit_tsources {true,false}`
		`+ Should we commit translated sources?`
		`+ --component COMPONENT`
		`+ Antora component to convert, all if unset`

		`- First, create an empty directory for the output and run the script:`
		+ The `buid.py` script will create three folders:

		- ```
		`- $ mkdir translation-repos`
		`- $ ./all-src-to-pot.sh $(pwd)/translation-repos`
		- ```
		`-`
		- Now you have local copies of the translation repos under `./translation-repos`, with the updated POT files. The next step is to push the changes to each repo. Go to the directory, and list all the repos:
		`-`
		- ```
		`- $ cd translation-repos`
		`- $ ls`
		`- docs fedora fedora-install-guide fedora-release-notes`
		- ```
		`-`
		`- Then go to each repo, and push the changes. For example:`
		`-`
		- ```
		`- $ cd docs`
		`- $ git add .`
		`- $ git commit -m "refresh POT files"`
		`- $ git push`
		`- $ cd ..`
		- ```
		`-`
		`- Repeat this for each repo. And you're done.`
		`-`
		`- ### Retrieving the translated sources`
		`-`
		- The `$ ./all-po-to-src.sh` holds a list of repositories to be translated and a list of target languages.
		`- The output is a local copy of the translated sources repo, updated with the newly generated adoc files.`
		`-`
		`- [See the structure of the translated sources](https://pagure.io/fedora-docs/design-documents/blob/master/f/po-pot-repo-structure.md).`
		`-`
		`- First, create a directory for the output and run the script:`
		`-`
		- ```
		`- $ mkdir translated-sources`
		`- $ ./all-po-to-src.sh $(pwd)/translated-sources`
		- ```
		`+ * $(pwd)/sources — Fedora Docs sources`
		`+ * $(pwd)/l10n — Translation repos`
		`+ * $(pwd)/translated-sources — Translated sources`

		- Now you have a local copy of the translated sources repo, right in `./translated-sources`. To push the changes:
		`+ ### Publish translated sources`

		```
		`$ cd translated-sources`
		`- $ git add .`
		`- $ git commit -m "update translated source"`
		`+ $ git add.`
		`+ $ git commit -m "automatic update of translated content"`
		`$ git push`
		`- $ cd ..`
		```

		`### Rebuilding the translated website`
		`@@ -89,5 +62,5 @@`
		`$ ./preview.sh`
		```

		`- Now you should be able to see, for example, the French site, on [http://localhost:8080/fr-FR/](http://localhost:8080/fr-FR/).`
		`+ Now you should be able to see, for example, the French site, on [http://localhost:8080/fr/](http://localhost:8080/fr/).`

build.py

file modified

+48 -23

		`@@ -10,24 +10,24 @@`

		`import yaml`

		`- COMMIT_CHANGES = 'false'`
		`SELECTED_COMPONENT = None`

		`def main():`
		`"""Handle params"""`
		`- global COMMIT_CHANGES, SELECTED_COMPONENT`
		`+ global SELECTED_COMPONENT`

		`parser = argparse.ArgumentParser(`
		- description="Calls `./po-to-src.sh` for all source repos")
		+ description="Calls `./src-to-pot.sh` and `./po-to-src.sh` for all source repos")
		`parser.add_argument("--clone_sources", choices=['true', 'false'], required=True,`
		`- help="Should we clone or update source or use cache?")`
		`+ help="Should we clone/update sources or use local content?")`
		`parser.add_argument("--commit_l10n", choices=['true', 'false'], required=True,`
		`- help="Should we clone or update source or use cache?")`
		`+ help="Should we commit changes on l10n repos?")`
		`+ parser.add_argument("--commit_tsources", choices=['true', 'false'], required=True,`
		`+ help="Should we commit translated sources?")`
		`parser.add_argument("--component", required=False,`
		`- help="Antora component, all if unset")`
		`+ help="Antora component to convert, all if unset")`
		`args = parser.parse_args()`

		`- COMMIT_CHANGES = args.commit_l10n`
		`SELECTED_COMPONENT = args.component`

		`source_dir = os.path.dirname(os.path.abspath(__file__)) + "/sources/"`
		`@@ -44,6 +44,12 @@`
		`clone_translated_source()`
		`make_component_lists()`

		`+ if args.commit_l10n == "true":`
		`+ commit_l10n_repos()`
		`+`
		`+ if args.commit_tsources == "true":`
		`+ commit_translated_sources()`
		`+`
		`def call_src_to_pot(url, branch='master', basedir='/', module="ROOT"):`
		`"""Call src-to-ot.sh to convert english source (adoc) content to pot files"""`
		`print("call_src_to_pot: "+url +" b:"+ branch +" bd:"+ basedir +" m:"+ module)`
		`@@ -56,10 +62,10 @@`
		`"""Call po-to-src.sh to convert po files to translated sources (adoc)"""`
		`print("call_po_to_src: "+url +" b:"+ branch +" bd:"+ basedir +" m:"+ module)`
		`output = os.path.dirname(os.path.abspath(__file__)) + "/translated-sources/"`
		`- for lang in ['fr', 'cs', 'ja']:`
		`- subprocess.run(['./po-to-src.sh', "--src-branch", branch,`
		`- "--src-basedir", basedir, "--module", module, lang, url,`
		`- output], check=True)`
		`+`
		`+ subprocess.run(['./po-to-src.sh', "--src-branch", branch,`
		`+ "--src-basedir", basedir, "--module", module, url,`
		`+ output], check=True)`

		`def clone_source(doc_url, branch="master"):`
		`"""Clone english sources in sources folder (update content if existing)"""`
		`@@ -73,10 +79,11 @@`
		`else:`
		`subprocess.run(["git", "clone", "-b", branch, doc_url, repo_dir], check=True)`

		`- def clone_l10n(url):`
		`+ def clone_l10n(repo_name):`
		`"""Clone localization repos in l10n folder (update content if existing)"""`
		`- print("* clone_l10n: " + url)`
		`- repo_name = url.rsplit('/', 1)[-1].replace('.git', '')`
		`+ print("* clone_l10n: " + repo_name)`
		`+`
		`+ url = "https://pagure.io/fedora-l10n/" + repo_name`
		`repo_dir = os.path.dirname(os.path.abspath(__file__)) + "/l10n/" + repo_name`

		`if os.path.exists(repo_dir):`
		`@@ -98,16 +105,35 @@`
		`else:`
		`subprocess.run(["git", "clone", url, repo_dir], check=True)`

		`- def commit_l10n(url):`
		`- """Commit and push changes of a localization repo"""`
		`- global COMMIT_CHANGES`
		`+ def commit_translated_source():`
		`+ """Commit translated sources"""`
		`+ print("* commit_translated_source")`
		`+ repo_dir = os.path.dirname(os.path.abspath(__file__)) + "/translated-sources/"`

		`- if COMMIT_CHANGES != "true":`
		`- return`
		`+ subprocess.run(['git', 'add', '.'], check=True, cwd=repo_dir)`
		`+`
		`+ subprocess.run(["git", "commit", "-m", "automatic update of translated content"],`
		`+ check=True, cwd=repo_dir)`
		`+`
		`+ subprocess.run(['git', 'push'], check=True, cwd=repo_dir)`
		`+`
		`+`
		`+ def commit_l10n_repos():`
		`+ """Commit and push changes of localization repositories"""`
		`+`
		`+ print("* commit_l10n_repos")`
		`+ repo_dir = os.path.dirname(os.path.abspath(__file__)) + "/l10n/"`
		`+`
		`+ # commit each folders of the l10n path`
		`+ [commit_l10n(r) for r in next(os.walk(repo_dir))[1]]`
		`+`
		`+`
		`+ def commit_l10n(repo_name):`
		`+ """Commit and push changes of a localization repo"""`

		`- print("* commit_l10n: " + url)`
		`+ print("* commit_l10n: " + repo_name)`

		`- repo_name = url.rsplit('/', 1)[-1].replace('.git', '')`
		`+ url = "https://pagure.io/fedora-l10n/" + repo_name`
		`repo_dir = os.path.dirname(os.path.abspath(__file__)) + "/l10n/" + repo_name`

		`subprocess.run(['git', 'add', '.'], check=True, cwd=repo_dir)`
		`@@ -177,9 +203,8 @@`
		`target_repo_name = component`
		`else:`
		`target_repo_name = component + "-" + module`
		`- clone_l10n("https://pagure.io/fedora-l10n/"+target_repo_name)`
		`+ clone_l10n(target_repo_name)`
		`call_src_to_pot(url=comp_dir, basedir=basedir, branch=branch, module=module)`
		`- commit_l10n("https://pagure.io/fedora-l10n/"+target_repo_name)`
		`call_po_to_src(url=comp_dir, basedir=basedir, branch=branch, module=module)`

		`def make_component_lists():`

po-to-src.sh

file modified

+76 -76

		`@@ -9,7 +9,6 @@`
		`echo "Arguments:"`
		`echo " REPO_URL Source repository URL."`
		`echo " OUTPUT_DIR Path to an output directory."`
		`- echo " LOCALE What language to work with. Such as 'fr-FR'."`
		`echo ""`
		`echo "Parametres:"`
		`echo " Generic:"`
		`@@ -51,16 +50,6 @@`
		`esac`
		`done`

		`- # Need LOCALE`
		`- if [ "$1" != "" ]; then`
		`- language=$1`
		`- shift`
		`- else`
		`- usage`
		`- echo "Please specify LOCALE, REPO_URL and TARGET_DIR"`
		`- exit 1`
		`- fi`
		`-`
		`# Need REPO_URL`
		`if [ "$1" != "" ]; then`
		`src_repo=$1`
		`@@ -98,6 +87,10 @@`
		`src_basedir=""`
		`fi`

		`+ echo "*****************"`
		`+ echo "* po-to-src"`
		`+ echo "*****************"`
		`+`
		`tmp_english_src=$(mktemp -d)`

		`delete_tmp() {`
		`@@ -204,76 +197,83 @@`
		`# Missing po for .adoc: the .adoc just get s copied`
		`#`
		`# TODO: stretch idea: if the translations don't exist, I could just copy`
		`- # the english sources over... this could simplify automation`
		`-`
		`- #po_pot_module_dir="$output_dir/$po_pot_repo_name/pot/$version"`
		`- po_pot_module_dir="$tmp_po_pot/po/$language/$version"`
		`- output_module_dir="$output_dir/$language/$component/$version/modules/$module"`
		`-`
		`- # Make sure the language exists in the PO/POT repo`
		`- # if [ ! -d "$tmp_po_pot/po/$language" ] ; then`
		`- # echo "Error. Locale $language doesn't exist in the PO/POT repo."`
		`- # delete_tmp`
		`- # exit 1`
		`- # fi`
		`+ # the english sources over... this could simplify automation`
		`+`
		`+ for lang_folder in "$tmp_po_pot/po/"* ; do`
		`+ [[ -d "$lang_folder" ]] \|\| break # if no result`
		`+`
		`+ language=$(basename $lang_folder)`
		`+ echo "*****************"`
		`+ echo "* LANG: $language"`
		`+ echo "*****************"`
		`+ #po_pot_module_dir="$output_dir/$po_pot_repo_name/pot/$version"`
		`+ po_pot_module_dir="$tmp_po_pot/po/$language/$version"`
		`+ output_module_dir="$output_dir/$language/$component/$version/modules/$module"`
		`+`
		`+ # Confert all .adoc files to .pot`
		`+ pushd "$tmp_english_src/$src_basedir/modules/$module" > /dev/null \|\| exit`
		`+`
		`+ for source_adoc_file in $(find . -name '*.adoc'); do`
		`+ po_file="$po_pot_module_dir/$(dirname "$source_adoc_file")/$(basename -s .adoc "$source_adoc_file").po"`
		`+ target_adoc_file="$output_module_dir/$(dirname "$source_adoc_file")/$(basename -s .adoc "$source_adoc_file").adoc"`
		`+`
		`+ format="asciidoc"`
		`+ # nav.adoc is a bullet point list, it isn't really asciidoc`
		`+ if [ "$(basename -s .adoc "$source_adoc_file")" = "nav" ] ; then`
		`+ format="text"`
		`+ fi`

		`- # Confert all .adoc files to .pot`
		`- pushd "$tmp_english_src/$src_basedir/modules/$module" > /dev/null \|\| exit`
		`-`
		`- for source_adoc_file in $(find . -name '*.adoc'); do`
		`- po_file="$po_pot_module_dir/$(dirname "$source_adoc_file")/$(basename -s .adoc "$source_adoc_file").po"`
		`- target_adoc_file="$output_module_dir/$(dirname "$source_adoc_file")/$(basename -s .adoc "$source_adoc_file").adoc"`
		`- if [ -f "$po_file" ] ; then`
		`- # TODO: convert file`
		`- if ! po4a-translate \`
		`- --format asciidoc \`
		`- --master "$source_adoc_file" \`
		`- --master-charset "UTF-8" \`
		`- --po "$po_file" \`
		`- --localized "$target_adoc_file" \`
		`- --localized-charset "UTF-8" \`
		`- --keep 0`
		`- then`
		`- echo "Error converting a PO file: $po_file"`
		`- delete_tmp`
		`- exit 1`
		`+ if [ -f "$po_file" ] ; then`
		`+ # TODO: convert file`
		`+ if ! po4a-translate \`
		`+ --format "$format" \`
		`+ --master "$source_adoc_file" \`
		`+ --master-charset "UTF-8" \`
		`+ --po "$po_file" \`
		`+ --localized "$target_adoc_file" \`
		`+ --localized-charset "UTF-8" \`
		`+ --keep 0`
		`+ then`
		`+ echo "Error converting a PO file: $po_file"`
		`+ delete_tmp`
		`+ exit 1`
		`+ fi`
		`+ else`
		`+ # "Warning: The following file is in the source, but not in the translations repo:"`
		`+ mkdir -p "$(dirname "$target_adoc_file")"`
		`+ cp "$source_adoc_file" "$target_adoc_file"`
		`fi`
		`- else`
		`- # "Warning: The following file is in the source, but not in the translations repo:"`
		`- mkdir -p "$(dirname "$target_adoc_file")"`
		`- cp "$source_adoc_file" "$target_adoc_file"`
		`- fi`
		`- done`
		`+ done`

		`- popd > /dev/null \|\| exit`
		`+ popd > /dev/null \|\| exit`


		`- # Convert the antora.yml to antora.yml.pot`
		`- # if this is the main module`
		`- if [ "$module" = "ROOT" ]; then`
		`- pushd "$tmp_english_src/$src_basedir" > /dev/null \|\| exit`
		`-`
		`- source_antora_yml="$tmp_english_src/$src_basedir/antora.yml"`
		`- po_antora_yml="$tmp_po_pot/po/$language/$version/antora.yml.po"`
		`- target_antora_yml="$output_dir/$language/$component/$version/antora.yml"`
		`-`
		`- if [ -f "$po_antora_yml" ] ; then`
		`- po4a-translate \`
		`- --format yaml \`
		`- --option keys="title" \`
		`- --master "$source_antora_yml" \`
		`- --master-charset "UTF-8" \`
		`- --po "$po_antora_yml" \`
		`- --localized "$target_antora_yml" \`
		`- --localized-charset "UTF-8" \`
		`- --keep 0`
		`- else`
		`- # "Warning: There should be an antora.yml.po, but there isn't."`
		`- mkdir -p "$(dirname "$target_antora_yml")"`
		`- cp "$source_antora_yml" "$target_antora_yml"`
		`+ # Convert the antora.yml to antora.yml.pot`
		`+ # if this is the main module`
		`+ if [ "$module" = "ROOT" ]; then`
		`+ pushd "$tmp_english_src/$src_basedir" > /dev/null \|\| exit`
		`+`
		`+ source_antora_yml="$tmp_english_src/$src_basedir/antora.yml"`
		`+ po_antora_yml="$tmp_po_pot/po/$language/$version/antora.yml.po"`
		`+ target_antora_yml="$output_dir/$language/$component/$version/antora.yml"`
		`+`
		`+ if [ -f "$po_antora_yml" ] ; then`
		`+ po4a-translate \`
		`+ --format yaml \`
		`+ --option keys="title" \`
		`+ --master "$source_antora_yml" \`
		`+ --master-charset "UTF-8" \`
		`+ --po "$po_antora_yml" \`
		`+ --localized "$target_antora_yml" \`
		`+ --localized-charset "UTF-8" \`
		`+ --keep 0`
		`+ else`
		`+ # "Warning: There should be an antora.yml.po, but there isn't."`
		`+ mkdir -p "$(dirname "$target_antora_yml")"`
		`+ cp "$source_antora_yml" "$target_antora_yml"`
		`+ fi`
		`+ popd > /dev/null \|\| exit`
		`fi`
		`- popd > /dev/null \|\| exit`
		`- fi`
		`-`
		`+ done`

		`delete_tmp`

src-to-pot.sh

file modified

+13 -2

		`@@ -214,10 +214,15 @@`
		`do`
		`pot_file="$module_dir/$(dirname "$adoc_file")/$(basename -s .adoc "$adoc_file").pot"`

		`+ format="asciidoc"`
		`+ # nav.adoc is a bullet point list, it isn't really asciidoc`
		`+ if [ "$(basename -s .adoc "$adoc_file")" = "nav" ] ; then`
		`+ format="text"`
		`+ fi`
		`# generate new POT files`

		`if ! po4a-gettextize \`
		`- --format asciidoc \`
		`+ --format "$format" \`
		`--master "$adoc_file" \`
		`--master-charset "UTF-8" \`
		`--localized-charset "UTF-8" \`
		`@@ -262,8 +267,14 @@`
		`# po4a-updatepo would be angry otherwise`
		`sed -i 's/Content-Type: text\/plain; charset=CHARSET/Content-Type: text\/plain; charset=UTF-8/g' "$po_file"`

		`+ format="asciidoc"`
		`+ # nav.adoc is a bullet point list, it isn't really asciidoc`
		`+ if [ "$(basename -s .adoc "$adoc_file")" = "nav" ] ; then`
		`+ format="text"`
		`+ fi`
		`+`
		`if ! po4a-updatepo \`
		`- --format asciidoc \`
		`+ --format "$format" \`
		`--master "$adoc_file" \`
		`--master-charset "UTF-8" \`
		`--po "$po_file" ; then`