The a11y team tracks issues here : https://gitlab.com/fedora/dei/a11y/-/issues

It would certainly be nice to have a11y guaranteed to be always working. Regarding our Quality team, I have the following questions/concerns:
1. What exactly do you envision to be covered by the new release criteria? The test cases above cover many different areas. All of them? Also, where do you want to draw the line between working and non-working? (E.g. which apps/system UI needs to be functional with the screen reader?). And should this only affect a certain desktop? Non-desktop editions as well? The installer? Initial setup?
2. Will general testers be able to test this without any special knowledge or hardware? At least brltty seems to require a Braille display (and knowing the alphabet). But even in other test cases, are the steps easy enough to follow even for people who have no experience with a11y tooling? I am concerned whether we'll have people who can test this. If people experienced with a11y are necessary, who is going to test it?
3. This seems to be mostly a manual job, without any option to automate it. How much time is this realistically going to cost us for each test compose?

This will obviously also require buy-in from developers, confirming that they have people who can fix issues in these tools in time.

Similarly to our recent discussion about new release blocking criteria (in that case, KDE blocking on aarch64), I wonder if it makes sense to try with a gradual rollout first - i.e. prepare everything but the criteria themselves (mostly write/polish up test cases and include them in our validation matrices), and see if we can get sufficient test coverage through the upcoming release cycle.

accerciser;https://fedoraproject.org/wiki/QA:Testcase_accerciser

It's also worth looking at accessibility-inspector: https://apps.kde.org/accessibilityinspector/

So the workstation tech spec does cover a11y: https://fedoraproject.org/wiki/Workstation/Technical_Specification#Accessibility

"The accessibility support in the workstation includes a screen reader, a high-contrast theme and a zoom capability, amongst others. The screen reading is provided through orca, which runs as a session service and requires the at-spi infrastructure. Applications are expected to provide suitable information to the screen reader via the toolkit's accessibility support. Applications are also expected to work acceptably in the high-contrast theme. The zoom is implemented in the desktop shell and does not need any application support."

...but we never wrote it into the release criteria. So we do have backing to do this, but...we need to know that the desktop team is on board with this, because we would need their commitment to fix problems found by these tests as a high priority (as @kparal said). Let's tag @catanzaro for that.

I would say we should try to stick to the scope defined in the tech spec, so cover screen reading, and desktop-level zoom and high-contrast themes. Zoom and high-contrast themes can be tested by openQA. In theory openQA could test screen reading too - it has a crazy/genius system for asserting sounds, by just taking a waveform graph of sound played on the VM's sound adapter and comparing it like a screenshot - but we've never used that mechanism before and don't know if it's really reliable enough to be used for this (@lruzicka , did you say you'd experimented with it recently? I forget). If not, that would have to be a manual test, which as @kparal says, requires resources.

When I talked to the openSUSE folks at oSC in June, they were fairly confident that screen reading can be tested through the mechanism you described (I'd asked because I wanted to have Fedora KDE a11y tests too). Santiago Zarate gave me a brief outline of the capability, and then I forgot to bring it up with you when I got back home. :sweat_smile:

We shouldn't block on accerciser since it's often broken. There is a modern replacement, though: https://gitlab.gnome.org/feaneron/elevado

Blocking on basic a11y functionality seems fine to me.

So we do have backing to do this, but...we need to know that the desktop team is on board with this, because we would need their commitment to fix problems found by these tests as a high priority

To the extent that we're good at fixing anything as high priority (we are not): yes, ack.