#6044 Download server changed behaviour
Closed: Invalid 6 years ago Opened 6 years ago by mkeir.


The only thing I can think of is we moved 04 and 05 to only be accessable from tier1 mirrors.

Otherwise I cannot think of a reason nginx would fail and apache work.

Do you have any errors in your nginx logs?

error.log

An example from a debug session is attached.
The issue is not reproducible on that server at the moment as I have switched it to use the Apache proxies as the source.

The download servers in PNQ, PEK2, TLV and BLR are however experiencing this issue and can be attempted as reproducers.

The source server we are using is dl01.fedoraproject.org, not 04 or 05 mentioned.

Requesting an update for this issue please.

Looking through your debug log, the nginx client sends "Host: fedora_source".
That is not a hostname that our download servers know to answer.

That's an upstream reference.
This had been working since November.

[root@download ~]# hostname
download.eng.pek2.redhat.com
[root@download ~]# more /etc/nginx/conf.d/revproxy.conf 
#define the upstream to be used later by proxy_pass dierctive, upstream  is a complex and powerfull option.
#http://nginx.org/en/docs/http/ngx_http_upstream_module.html
upstream download_source  {
# print server to pull Red Hat content from depending on what revproxy_source
# is set to in the host def file default or site config
server download-node-02.eng.bos.redhat.com;
}

#top server defintion here we define where we will place cahce and inside server defintion we will host
# our locations , thi is basicaly a bind and defintions. Inside eval cyccle we are giving all the DOM names of curent revproxy
upstream fedora_source  {
# print server to pull Fedora content from depending on what fedora_source
# is set to in the host def file default or site config
server dl01.fedoraproject.org;
}
upstream mrepo_source  {
  server download.devel.redhat.com;
}
server {
  listen 80;

    server_name download.eng.pek2.redhat.com; 
  access_log  /var/log/nginx/access.log revproxy;
  set $no_cache 0;
  location / {
    if ($request_uri ~* "\.(xml|xml.gz|xml.asc|sqlite|sqlite.bz2|iso)$")
      {
        set $no_cache 1;
      }
    proxy_cache_bypass      $no_cache  ;
    proxy_no_cache          $no_cache ;
    proxy_pass              http://download_source ;
    proxy_set_header        Host $host;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-Server $host;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    add_header              Custom proxied;
    add_header              X-Cached $upstream_cache_status;
    proxy_cache             default;
    proxy_cache_use_stale   error timeout invalid_header updating http_500 http_502 http_503 http_504;
    proxy_cache_key         $host$uri ;
    proxy_cache_valid       200 1h;
    proxy_cache_valid       302 30m;
    proxy_cache_valid       301     1h;
  }
  # do not proxy /gsync /cgi-bin /errors those are local stuff.
  location /pub {
    expires -1 ;
    add_header No-X-Cache-msg "local gsync data /mnt/redhat" ;
    autoindex on ;
    alias /mnt/redhat ;
  }
 location /pub/fedora {
    expires -1;
    autoindex on;
    alias /mnt/redhat/fedora;
    try_files $uri $uri/ @fedora;
  }

  location @fedora {
    proxy_pass  http://fedora_source;
  }

  location /pub/rhel/released {
    expires -1;
    autoindex on;
    alias /mnt/redhat/rhel/released;
    try_files $uri $uri/ @released;
  }

  location @released {
    rewrite ^/pub/rhel/(.*)$ /$1 break;
    proxy_pass  http://download_source;
  }

  location /rel-eng {
    expires -1;
    autoindex on;
    alias /mnt/redhat/rel-eng;
    try_files $uri $uri/ @rel-eng;
  }

  location @rel-eng {
    proxy_pass  http://download_source;
  }

  location /sysrepos {
    proxy_pass  http://mrepo_source;
  }

}

Could you for grins add "proxy_set_header Host dl.fedoraproject.org;" on line 69, so within the "location @fedora" block?

Basically, as I said before, you are sending a Host header of "fedora_source", which is almost guaranteed to not work.
Also, I note that you only have dl01.fedoraproject.org in your upstream reference.
Note that we do NOT announce outages when we reboot or reinstall dl* nodes or when they break, because they should not cause outages for any users, since we will just pull them out of roulation.
You should seriously consider adding the other mirrors there if you want to continue with this way of doing things.

More ideally however, you would not need this, and just use the Red Hat internal mirror infrastructure to get to Fedora packages.

The hosts in question are the Red Hat internal mirror infrastructure. That's what the changes have broken.

Well, except we have not made any changes to dl01.fedoraproject.org that I can think of aside from normal updates and reboots.

Did you get a chance to try the suggestion @puiterwijk had?

What changes? The last change that I can see in the download httpd files is from July of 2015. There were changes for rsync in April but this is not an rsync problem from what I can tell.

Can I get the ip address you are trying from?

Per email and posted reply, both of which have not turned up here, please find the following information.

"This issue can be closed, I have found a workaround to restore the global Red Hat proxy/mirror infrastructure.

location @fedora {
proxy_set_header Host dl.fedoraproject.org;
proxy_pass http://dl.fedoraproject.org;
proxy_redirect http://dl.fedoraproject.org/pub/fedora /pub/fedora;
}
"

Metadata Update from @puiterwijk:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

6 years ago

Login to comment on this ticket.

Metadata
Attachments 1
Attached 6 years ago View Comment