#9126 Fedora Docs translation scripts- obsolete content on homepage
Opened a month ago by jibecfed. Modified 21 hours ago

The issue, this page contain obsolete content: https://docs.fedoraproject.org/fr/docs/

Its real content is in: https://pagure.io/fedora-docs/translated-sources/blob/master/f/fr/docs/master/modules/ROOT/pages/index.adoc

Source file talks about Fedora 32: https://pagure.io/fedora-docs/pages/blob/master/f/homepage/modules/ROOT/pages/index.adoc

And translation file contain the Fedora 32 translation: https://pagure.io/fedora-docs-l10n/docs/blob/master/f/po/fr/master/pages/index.po#_37

The script supposed to update the content is ran on a daily basis in: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/fedora-docs/translation/files/docs-translation-update#_13

To test the behaviour and make faster test, just add --component docs, so that only this component is build.

Test process:

  • clone https://pagure.io/fedora-docs/translations-scripts/
  • run ./build.py --clone_sources true --commit_l10n false --commit_tsources false --component docs --clone-po4a
  • search for Fedora 32 in this file:translated-sources/fr/docs/master/modules/ROOT/pages/index.html

It works on my computer, but I can't understand why doesn't it work when executed automatically.
Please note this isn't a recent issue, it never worked and it took me time to notice it and open the bug.

My assumption is that for the homepage, we use XML parser and that po4a uses perl, and we may have an incompatibility somewhere.
I have no idea how to locally reproduce the ansible environment. My first try would be to take a more recent platform.


Metadata Update from @pingou:
- Issue tagged with: dev

a month ago

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, medium-gain, medium-trouble

a month ago

I've reproduce this issue, and it seems to be related to the YAML::Tiny perl module complaining about malformed antora.yml (this one more especially https://pagure.io/fedora-docs/pages/blob/master/f/homepage/antora.yml).
And it's not completely wrong, as the ROOT::index.adoc part should be quoted here to avoid colons messing with the format.
I've not tried yet, but since @jibecfed is not affected, I suspect a more up-to-date library might be a little bit more flexible in this regards.

EDIT: Confirmed. The YAML::Tiny module shipped in rhel7 (1.51) can't read the antora.yml file, the new one directly from cpan (1.73) can.

So the solution is to quote the file where it should be quoted ? (or upgrade the server to EL 8)

Exactly. Another solution would be to move this app to openshift.

Le lun. 20 juil. 2020 =C3=A0 11:59, Michael S pagure@pagure.io a =C3=A9cr=
it :

misc added a new comment to an issue you are following:
So the solution is to quote the file where it should be quoted ? (or upgrade the server to EL 8)

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-infrastructure/issue/9126

sorry, but adding the quote had no impact

When i try to build locally on RHEL 7, I get it updated (index.adoc) no matter the type of quote I use (as I wondered about double quote vs single quote). Maybe the change was right but not reflected in prod for another reason ?

@darknao, since you can reproduce, can you tell if the quote change by @jibecfed work fine for you ?

The quote fixed the initial issue being the index.adoc not updated.
Now we have a second one, that the push to git is not working for some reason.

the pre commit hook maybe?

found it:
in build.py :

shutil.copytree(os.path.join(dir_path, '.githooks'), os.path.join(repo_dir, '.githooks'), dirs_exist_ok=True)

the dirs_exist_ok is added in python3.8, we are in python3.6.

Here is a patch proposal, if that can help.
0001-Python36-compatibility.patch

This remove the use of dirs_exist_ok with shutil.copytree and remove any local changes to your local cloned repositories so you can copy your githook back without errors.
The git reset can also be seen as a security measure as I assume there shouldn't be any local uncommitted changes at this point.

i "applied" the patch and it failed because of an existing folder=2E

note: did an ugly commit yesterday because git complained abouy you patch =
when doing git patch --check

i won't have access to computer for the next few days=2E

Misc, you have commit access on the repository=2E i would appreciate if yo=
u may have the opportunity to give it a look=2E

in addition, it looks like the script is executed on sundries01 and sundri=
es02 because i get two emails=2E
is it because of some migration or a real issue?

ok my bad, I was mistakenly thinking there was a .gitignore rules for the githook directory, so my git reset doesn't achieve the intended goal here.

After a more thoroughly reading of your script, there is a few things to fix.
If you need to overwrite .githook, then you may want to use distutil.dir_util.copy_tree instead.

Your pre-commit hook is not working for at least two reasons.
There is a syntax error line 26 : (("1" + "2")) doesn't work in bash4, ((1 + 2)) does.
This hook is also never called anyway. The usual hook directory is .git/hooks, and the core.hookspath config is only working since git 2.9 (el7 use git 1.8).
That is probably a good thing actually, since your hook do a git rm of your file, which completely remove it from the repo. I think you may want to only unstage it instead with git reset or simply abort the commit.

Most of these errors are related to rhel7 binaries. I can only recommend you to set up a dev/test environment using el7, if you can.
Moving this app to openshift may also be a good idea, and give you more control over the running environment.
I can probably help you with that, if you want.

0001-use-distutils.dir_util-instead-of-shutil-for-recursi.patch

And as of sundries02, I think that is a real issue too.

Openshift was the plan (and ideally, split that per repo so we do not wait several, but can schedule them and/or update them at different speed), but sundries was easier to do. I was waiting for the DC move to be officially over before working on that, but feel free to do it.

For the 2 sundries, I guess there was a double deployment on nasible due to the main sundries being down or something. It should be fixable in ansible without too much fuzz.

Ok, so I here is my proposal:
I can make a PR on your translations-scripts repo with a new Dockerfile that will be used by openshift, among a few fixes on your pre-commit hook & build.py.
Then another PR on ansible repo to create the associated openshift-app.
I will probably generate a new ssh key pair, so I'll send you the new pub key at this time.
Once this is done and working, we'll remove the scripts from sundries 1&2.
How does that sound to you ?

The ssh key should be stored in a secret on openshift, IMHO. The way I did in the past was to generate it and display when running the script, I guess it coud be more elegant. Alternatively, we can also just ask to a sysadmin to move the existing one in a openshift secret and that's it.

I think the key should be somewhere like in /srv/private so it can be easily used by ansible to generate the openshift secret.

Login to comment on this ticket.

Metadata
Boards 1
dev Status: Backlog
Related Pull Requests
  • #10 Merged 14 days ago