fedora-infrastructure

#7335 Make retrace.fp.o run behind proxies
Closed: Will Not/Can Not fix by smooge. Opened by clime.

Closed: Will Not/Can Not fix
Reopen Issue
  • Describe what you need us to do:
    I would like to make FAF service, which runs on retrace.fedoraproject.org run behind proxies.

There are currently two machines deployed with FAF, retrace01.fedoraproject.org and retrace02.fedoraproject.org but only retrace01.fedoraproject.org is used per DNS records.

retrace         IN    CNAME retrace01

It seems that that there are stubs for the HA setup here and there, e.g. from roles/haproxy/templates/haproxy.cfg:

backend retrace-backend
    balance hdr(appserver)
    server retrace02.qa.fedoraproject.org retrace02.qa.fedoraproject.org:80 check inter 10s rise 1 fall 2

or in playbooks/include/proxies-websites.yml:

- role: httpd/website
  site_name: retrace.fedoraproject.org
  server_aliases: [retrace.stg.fedoraproject.org]
  sslonly: true
  cert_name: "{{wildcard_cert_name}}"
  when: env == "staging"

but I don't need full HA setup immediatelly, I would just like to make retrace.fp.o run behind proxies for now as the first step. it would be cool if we could prepare the OpenVPN private keys and the static urls for the retrace server (per https://docs.pagure.org/infra-docs/sysadmin-guide/sops/openvpn.html) so that everything is ready for HA and then we can just later switch the abrt/faf-local role to abrt/faf, which would involve migrating data to the new single PostgreSQL db.

  • When do you need this? (YYYY/MM/DD)

  • When is this no longer needed or useful? (YYYY/MM/DD)
    Always useful.

  • If we cannot complete your request, what is the impact?
    We would like to make steps towards HA setup for the FAF servers. Given that they are directly linked with people desktops, it's useful to have them (almost) always available.

retrace is on a network without vpn. This was done on purpose for several reasons.

  1. retrace was to be considered always 'hostile'. Because it is getting random 'crap' as the authors put it, they could not say it wasn't likely to be taken over at any time.
  2. The system is on the QA network and not reachable by the openvpn server. [This won't change.]

Next retrace is not a good candidate for the existing proxies.
1. They are not useful for uploading large amounts of data. They are useful for 'getting' large amounts of data. Past attempts to make them good for uploading large files ended up making the other parts bad or other problems.
2. They are not HA in the manner you are expecting. This is why some of the 'proxies' front ends only go to proxy01/proxy10 because latency for sending stuff to internet05 to PHX2 over vpn breaks things which assume near transit.

This doesn't mean we can not put some sort of proxies in front of the retrace servers, but they would need to be in Phoenix and they would need to be tuned for uploading large files versus downloading large files.

So based on what you are saying about the uploading, I think it might be worth considering to separate FAF service from retrace-service. Only retrace server needs to receive the uploaded coredumps. FAF doesn't need it so FAF could be placed behind the proxies if it makes the service actually run better, which should be further argued.

Some additional info about the services:

FAF only receives anonymous micro-reports from users to provide statistical analysis and linkage with other bug reporting sites.

Retrace server receives the uploaded coredumps to generate full backtraces from them (by using gdb) and it then returns the results back to users so that users can open a Bugzilla ticket with the generated backtrace info.

Those two parts (FAF and retrace) are essentially separate but they run on the same machine currently because both FAF and retrace need to have Fedora packages downloaded locally so that name resolving is possible.

FAF does the name resolving too but only just to determine function names from program offsets that are present in the micro-reports (it doesn't work with coredumps and it doesn't use gdb).

So maybe we could consider making two FAF server and one retrace server, all sharing the same volume with downloaded packages and the two FAF servers would e.g. be HA and have a shared DB.

Currently, everything (RETRACE+FAF+DB) runs on a single machine where the packages are also being downloaded. There is a machine retrace02.qa.fedoraproject.org, which is deployed but essentially unused right now.

Metadata Update from @bowlofeggs:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

@msuchy What would you like to do here?

I am not sure moving these services behind our proxies really gets us much. The proxy network is good at caching stuff close to users, but there's very little caching involved here.

Also, if we end up moving this to the cloud or our cloud network it would be directly contacted there.

We can close this.

Metadata Update from @smooge:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.