fedora-localization-statistics

Created a year ago
Maintained by jibecfed
Global statistics on translation levels of fedora products
Members 1
Jean-Baptiste Holcroft committed 5 months ago

fedora-localization-statistics

Global statistics on translation levels of fedora products

Requirements

dnf install translate-toolkit podman

Create needed folders

mkdir -p ./src.rpms/f30/ ./results/f30/
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

Run the scripts

Get package list

This step is for now manual, I took list of DNF packages from Koji:

Get the rpm sources

./download-f%%-srpm-in-container.sh where %% is the fedora version (30 or 31)

Downloading the file is done inside a container so we can produce stats even if using Fedora 29. This represents about 7 GB for Fedora 30 and takes some time.

Compute data

./build.py

The result will be in multiple files inside the results folder.

Produce stats

./build_stats.py

Applies data cleanups and enhancements (cldr name).

Produce map

./build_map.py

Agregate the data per language, then apply it on territories (it uses stats from CLDR with language per territory).

Output files

  • 0.error.language not in cldr.csv contains unknown languages (lines are removed)
  • 0.error.languages is numeric.csv contains numeric languages (lines are removed)
  • 0.error.lang with point.csv contains languages such as ".cp936" ".big5" (lines are removed)
  • 0.error.len(language).csv contains languages with more than three caracters (lines are removed)
  • 0.error.len(territory).csv contains territory with more than two caracters (lines are removed)
  • 0.error.no population for this language-territory couple.csv contains the list of language-territory couple where no language statistics exists (no impact on results)
  • 1.debug.lang.csv all lang (language + script + territory) values for debug (no impact on results)
  • 1.debug.language.csv all lang values for debug (no impact on results)
  • 1.debug.script.csv all script values for debug (no impact on results)
  • 1.debug.territory.csv all territory values for debug (no impact on results)
  • 1.debug.total message = 0.csv all lang values for debug (lines are removed)
  • 3.result.csv full results per package with source filename and standardized language code, script code and territory code
  • 4.0.cldr.csv language per territory as provided by CLDR
  • 4.1.results_per_language.csv message and words progress percentages per language
  • 4.1.results_per_language_ISO3.csv message and words progress percentages per language merged with "country code" database using ISO3166-1-Alpha-2 code
  • 4.2.cldr_and_results_full.csv language per territory as provided by CLDR merged with message and words progress percentages per language
  • 4.3.cldr_and_results_grouped.csv aggregation per territory of 4.2.cldr_and_results_full.csv, provides the territory, the number of languages, the population, the messages and words coverage.
  • 4.4.world_stats.csv merge results of 4.3.cldr_and_results_grouped.csv with country database and geojson data.

Informations

Data in CLDR-raw folder comes from https://github.com/unicode-org/cldr/blob/master/common/main/en.xml

Ideas

  1. CLDR supplementalData.xml: https://github.com/unicode-org/cldr/blob/master/common/supplemental/supplementalData.xml
    1. use territoryContainment to build geographic groups
    2. use languageData to detect default script
    3. use languageData to have basic stats about territories
    4. use territoryInfo to have advanced stats about territories
  2. CLDR supplementalMetadata.xml: https://github.com/unicode-org/cldr/blob/master/common/supplemental/supplementalMetadata.xml
    1. use the replacement values harmonize content
  3. CLDR likelySubtags.xml: https://github.com/unicode-org/cldr/blob/master/common/supplemental/likelySubtags.xml
    1. use the replacement advanced harmonization?
  4. CLDR languageInfo.xml: https://github.com/unicode-org/cldr/blob/master/common/supplemental/languageInfo.xml
    1. can we say if language is >= 90% close to another one, we can consider we propagate translation statistics?
  5. CLDR languageGroup.xml: https://github.com/unicode-org/cldr/blob/master/common/supplemental/languageGroup.xml
    1. what is it?

automatic calculation (group by territory + spoken percentage * spoken )

create stats: number of countries with official language > 50% and related population

create stats: number of languages impacting more than one official language

See also

AppData and Zanata statistics: https://github.com/Jibec/fedora-translation-statistics Transtats: https://transtats.fedoraproject.org/releases/