What ports are going to be open/accessible for the websocket server to work?

Its arbitrary, and right now I'm using 9919 in development for no good reason at all.

I think this stackoverflow Q/A might be useful -- http://stackoverflow.com/questions/2419346/can-nginx-be-used-as-a-reverse-proxy-for-a-backend-websocket-server

Ianweller found this one - http://jfarcand.wordpress.com/2011/10/06/configuring-haproxy-for-websocket/

Next step -- going to stand up busmon (the TG2 app) in app0{1,2}.stg. We'll use its failed websocket connection attempts to debug the forthcoming haproxy+websockets setup.

Open up https://apps.stg.fedoraproject.org/busmon/ to check it out.

If you do so in chromium, you'll see it happily try to connect to {{{ws://ws.fedoraproject.org:9919}}} over and over.

If you do so in firefox (with firebug open) you'll see it bomb out in javascript-land complaining that connecting to {{{ws://ws.fedoraproject.org:9919}}} is insecure: "websockets on a ssl page must be ssl websockets." We should be connecting to {{{wss://...}}}.

That's smart, Firefox. moksha-0.8.8-2 will set the protocol correctly.

Now that we have the TG2 app stood up, we can start working on the haproxy+websocket-server portion.

Our current (non-working) setup looks like:

{{{browser -> apache -> haproxy -> websocket-server}}}

But it looks like apache's proxy module throws away the required websockets headers.

http://serverfault.com/questions/290121/configuring-apache2-to-proxy-websocket

The two friendliest options look like:

1 - expose an haproxy port to the wider internet, say 10019.

2 - use mod_pywebsocket

I'm leaning towards the first. The documentation for mod_pywebsocket indicates that it is experimental and requires mod_python. http://code.google.com/p/pywebsocket/

Would there be any issue with exposing that haproxy port (and not the others)?

We can't use mod_pywebsocket... since it needs mod_python, and we are using mod_wsgi, and they conflict in the same apache instance. :(

So, options then fall under 1...

a) open that port on all the proxy boxes so we can do ws.fedoraproject.org:10019 or whatever. This will take some requests to folks to open it in phx2.

b) Setup a seperate box somewhere that has this port open, but then it needs to be able to use the vpn to communicate with phx2 resources. That may make it slow or not practical if it needs non routed traffic.
Also, may require firewall adjustment.

a) would expose haproxy more places, also, if we need more/different websockets services down the road, they each need their own port I guess? b) sounds like it might be slow and/or not work if we can't vpn.

Just a clarification: on a), the different future websocket services can actually share the same websocket server. That constrains the number of open ports we need down to just 1, which is a good thing.

Ah, thats good news.

Replying to [comment:7 kevin]:

We can't use mod_pywebsocket... since it needs mod_python, and we are using mod_wsgi, and they conflict in the same apache instance. :(

Apparently mod_wsgi 3.1 and mod_python 3.3.1 can coexist: https://bugzilla.redhat.com/show_bug.cgi?id=524120#c1

huh. We were seeing some strange behavior and now don't have anything that uses mod_python. ;)

I believe that originally we were porting away from mod_python to mod_wsgi because of compatibility (ex: the mirrormanager mirrorlist_server script). Recent ports (ex: koji) were motivated by the fact that mod_python is "dead" upstream (upstream claims they're just stable but, at the very least, there's a raft of shortcomings of mod_python architecturally that are never going to be fixed). (Although I seem to recall one place where we ported away from mod_python in the last two years that had a mod_python/mod_wsgi incompatibility.)

Dead upstream C code that interfaces with apache (which we're serving publically) is not something we want to be roped into maintaining. Especially now that we've finally succeeded in exorcising it from our infrastructure.

ok, here's the plan:

create a busgateway01.stg.fedoraproject.org instance.

request port 9939/tcp and 9940/tcp get forwarded in from 209.132.181.5 to the IP of the busgateway01.stg instance.

modify haproxy on proxy01.stg to listen on those ports and use busgateway01.stg as the backend.

The 9939/tcp port will be websockets.
The 9940/tcp port will be used for raw zmq messages.

Test and work out any problems.

Thoughts? Concerns?

ok. I have created the busgateway01.stg.phx2.fedoraproject.org instance.

I have also requested the port forwards.

Should we leave this ticket open? or close it out now that we have a plan we are implementing?

I'm fine closing it. Can you let me know once the port forwards are in place? I'll start in on configuring busmon and fedmsg then.