This ticket had assigned some Differential requests:
D589
D577
D582

This discussion started with D582 - I have created needles using needles editor (with default tags) and forgot to add changed tags when merging - the result was that KDE test started failing without reason (because KDE needle was unregistered).

So the first problem I had with this stuff is what I noted in an email today: openQA actually has a kinda hardwired default value of en_US for INSTLANG. I think, though, that setting it in the test suite settings for each test should work OK.

I'm not sure I really like the 'require all universal tags to be explicitly marked with every language tag' approach, it's just too English-centric, for me. I prefer the philosophy that all is the 'default'.

There is a way we can achieve that; it just requires a bit more cleverness in the unregister_except() function, which is why I didn't submit it at first. What we do is something like def unregister_except(prefix, value), which unregisters all needles which have any tag starting with prefix if they don't have a tag matching prefix-value. So e.g. if you do unregister_except('LANGUAGE', 'ENGLISH') it would unregister a needle with only the tag LANGUAGE-FRENCH, but not a needle with the tag LANGUAGE-ENGLISH, or a needle with no LANGUAGE- tag at all.

WDYT about that?

! In #617#7900, @adamwill wrote:
it's just too English-centric, for me. I prefer the philosophy that all is the 'default'.

On the other hand, English is THE DEFAULT for us. Even in test matrixes - they explicitly says when you are required to use different language than English.

There is a way we can achieve that; it just requires a bit more cleverness in the unregister_except() function, which is why I didn't submit it at first. What we do is something like def unregister_except(prefix, value), which unregisters all needles which have any tag starting with prefix if they don't have a tag matching prefix-value. So e.g. if you do unregister_except('LANGUAGE', 'ENGLISH') it would unregister a needle with only the tag LANGUAGE-FRENCH, but not a needle with the tag LANGUAGE-ENGLISH, or a needle with no LANGUAGE- tag at all.

WDYT about that?

I'm used to create needles using web needles editor, so I'd prefer solution what is easy to use and fool-proof for me ;-), but other than that,

{meme, src=neutralpresident, above="I HAVE NO STRONG FEELINGS", below="ONE WAY OR THE OTHER"}

OK, after talking with @kparal, I have yet another angle (hence apologizing here for completely ignoring D589 at the moment).

Looking at the testcases from the Internationalization and Localization section, the merit of the testcase is not really to tell that the strings are translated (even though it is a part of it), but IMHO the main reason for the testcases is to check that keymaps work. Looking at the needles and code in D577 we don't really check that while we have selected French, the french keymap is selected (OK, we do, as an side-effect, but not explicitly). We test that the type_string("encrypt_password")call sends the same sequence of characters in anaconda, and during boot.
Lucky enough for us, OpenQA actually sends keymaps, not chars, so we in fact do (as an side-effect) test if type_string("qwerty") called in the user-creation spoke, ends up "writing" the same string during login attempt. But since we don't really check the typed string really is azerty. So in the occasion, when the keymap selection does not work at all, we have a fake-pass.

On top of that, as I was told by @kparal, we don't really have criteria saying "Anaconda must be completely translated", we just say that "Anaconda must be able to show translated strings". So in a hypothetical situation, where only the main-hub is translated to French, but all the other strings are English, we should still be able to go through the install process via the english needles (where applicable). Yes, we can (and maybe even should) have some test for "is this translated?" on a string that is "fairly certainly translated".

Thus, I propose to going closer to what the testcases say, and do this:
# check the correct keymap is selected
# check that some string (arbitrary chosen) is translated
# get rid of unloading needles based on the contents of LANGUAGE (although I like the version in D589 much better than the D577 one)

1 and 2 can fairly easily be solved by
# if LANGUAGE is set, in main.pm load check_internationalization_and_translation test
# the check_internationalization_and_translation would
# go to the Language Support spoke
# type_string that "fairly certainly covers the differences in the keymaps" (e.g. qwerty is azerty in french, qwertz in czech, ... we could easily think of a good enough string to cover our needs...)
# we should also use this "clever string" as the password, later, I guess, so we really check that the keymaps work in all the places
# check that the screen shows the right characters, in a manner similar to the way we load partitioning tests (assert_screen('translation_'.GET_VAR('LANGUAGE')))
# also check that some string at the page is translated - easily done by selecting a label in the aforementioned translation_LANGUAGE needle.

This will give us an explicit check of the fact that keymaps work at all the relevant places, and that Anaconda can show translated strings. We would get rid of the need to (un)load needles on-demand, thus making the code less complicated. Keeping the needles tagged with LANGUAGE_XYZ is then not required, but I'd strongly support it, so we can use the information, should we ever really need to.

Thoughts? (and yes, I know you all hate me...)

After reading the test case, I agree with @jskladan - it looks like one missing translation that falls back to English is still OK in the terms of test case passing.

Overall...I'm not convinced, honestly. I prefer the tag approach as it is extensible. Your approach is more special-cased. Consider the case where openSUSE uses needle unregister: desktop tests. Maybe the first time you add a test that varies between desktops it seems easy to do assert_screen('some_screen'.get_var('DESKTOP'))....then you turn around three months later, realize you're doing it in 30 different places and that seems kinda silly, and re-write it to be more extensible. I'd like to skip to the rewrite bit. ;) Keeping code simple is nice, but at the same time I don't think it's a good idea to pretend we're not going to keep building out new tests. We do need some kind of degree of infrastructure for that. Note that my implementation is intentionally not at all Anaconda specific; it could be used for any other kind of l10n testing. And the code is really, honestly, not that freaking complicated, it's just a few lines of loops in perl's shitty syntax, heavily commented. It's not rocket science.

I did actually think about all the things you mention in designing the test. The user name is not 'qwerty' by accident, nor is the password 'weakpassword' by accident (anything with an 'a' in it differs between English and French layouts). It would be an improvement to add a needle checking the user name actually appears as 'azerty', yes, but that's easily done (we can add a test for it right after user creation, as the user name is displayed on the install hub).

One missing translation is still OK for the criteria, yes. I don't view openQA as solely a Criteria Enforcin' Machine, though. Especially for premium languages like French I'd like to check the translations more rigorously. There's no good reason any translation that currently exists should ever 'go missing'; translation status changes when the original string changes, and if any of these original strings changes that means we're going to have to re-do the needles anyway. So I think it's useful to know any time something that's currently translated suddenly stops being translated.

Note that with my design it's very easy to add "exceptions" for non-translated strings where appropriate: literally all you have to do is drop the LANGUAGE-english tag from the English needle, or (if you only want to add an exception for a specific language), add the LANGUAGE-whatever tag. This is already used - if you follow the French test through, there are several text needles that aren't translated (bootloader, console encryption password entry, login and post-install stuff), and so there is no _french variant needle for them and the English needle has no LANGUAGE tag.

I agree that tag approach is more extensible (and, let's say, better prepared for future). My main concern is that it should work when users are creating their needles using needles editor (as for example I do). I would take "sane defaults" approach - if someone is creating test for openQA, he is most likely creating English test. In that case, all the sane values should be selected. From what I understand, D589 enables this, but we should count with the fact that in the near future, there will be a lot of needles without LANGUAGE tag defined, because people will forgot (and I know it, it happened to me twice already). So I'm mostly okay with D589 (I think that it's better than current state), but still, I'm more of a "solve issues after they appear" type of a guy.

To me, current state would be okay IFF openQA would make it possible to specify what tags are selected in needles editor by default. In that case, we could specify that tag LANGUAGE-english is selected by default in every needle and whomever wants to create different language-specific test, he could edit this tag easily.

@adamwill I would still add "qwerty -> azerty" check needle to test, because while openQA isn't Criteria Enforcin' Machine, the fact that we mark that this test passed means that we should at least checked all required criteria.

Well, I mean, that was why I rejigged things in D589 so that it's perfectly fine for needles to have no LANGUAGE tag. They're simply treated as universal in that case.

Indeed I was planning to add the azerty test, yup, just didn't get to it yet.

OK, I landed this. As jsedlak says we can run with it for a while and adjust if we need to.