The new openQA code that's running on staging has a new 'developer mode' (replacing the old 'interactive mode') which relies on the user's browser being able to establish a websocket connection with the openQA server. For this to work, the request has to be proxied properly, but ATM I don't think it is, so if you visit any running test and go to the 'live view' tab, you'll just see a 'retrieving status' spinner forever; if you check the console you'll see the websocket request constantly failing to go through.
To fix this for my pet deployment, I had to add this line to the config for the relevant virtual host on my reverse proxy, above the catch-all ProxyPass line:
ProxyPass "/liveviewhandler/" "wss://openqa.happyassassin.net/liveviewhandler/" keepalive=On
i.e., basically, 'proxy requests for /liveviewhandler/ via wss, not https'. For infra this may need to be 'ws://' not 'wss://', I'm not sure.
YESTERDAY
NEVER
World War III. Or, I can't edit openQA tests interactively. One or the other. Who knows!
I think this may involve adding a new template in roles/httpd/reverseproxy/templates and adjusting playbooks/include/proxies-reverseproxy.yml, but I'm not 100% sure.
roles/httpd/reverseproxy/templates
playbooks/include/proxies-reverseproxy.yml
This is what I came up with:
diff --git a/playbooks/include/proxies-reverseproxy.yml b/playbooks/include/proxies-reverseproxy.yml index 6916c3993..f3275c7d4 100644 --- a/playbooks/include/proxies-reverseproxy.yml +++ b/playbooks/include/proxies-reverseproxy.yml @@ -322,6 +322,7 @@ remotepath: / localpath: / proxyurl: http://localhost:10044 + wsproxyurl: ws://localhost:10044 - role: httpd/reverseproxy website: apps.fedoraproject.org diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.openqa.conf b/roles/httpd/reverseproxy/templates/reversepassproxy.openqa.conf new file mode 100644 index 000000000..e35e6c8ec --- /dev/null +++ b/roles/httpd/reverseproxy/templates/reversepassproxy.openqa.conf @@ -0,0 +1,3 @@ +ProxyPass "/liveviewhandler/" "{{wsproxyurl}}/liveviewhandler/" +ProxyPass {{localpath}} {{proxyurl}}{{remotepath}} +ProxyPassReverse {{localpath}} {{proxyurl}}{{remotepath}}
not totally sure if that's right. Notably, I have no idea how this stuff somehow gets applied to staging? There is no specific entry for openqa.stg.fedoraproject.org in proxies-reverseproxy.yml , yet somehow it works...
Thanks very much to the awesome @puiterwijk we now have this working, using the 'balancer' proxying bits in the ansible infra scripts that generically handle all websockets traffic correctly.
The only remaining problem is that it's not working for aarch64 jobs, which I believe is because the aarch64 workers are on a different network. There's likely a firewall between the QA and arm networks that prevents the server (which is on the QA network) from establishing a websocket connection tot the worker (which is on the arm network).
Not sure how we should resolve that, let's look into it. It's not urgent, though, as we will usually be able to just use x86_64 or ppc64le jobs to edit needles; it's rare to need to use developer mode on a specific arch.
In order to open that firewall we would need to know what port(s) to ask to open...
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review)
That's the fun part :D There is one port per worker instance, so currently four ports per box.
The ports for a four-worker box are ['20013', '20023', '20033', '20043']. We need the server to be able to reach those four ports on each of the aarch64 worker boxes.
Those ports have been opened now. Please test and re-open if there's any issues remaining.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.