#12818 rdu2-cc to rdu3 datacenter move
Opened 2 months ago by kevin. Modified 20 days ago

We are going to be moving machines/services from our rdu2 'community cage' datacenter to rdu3 (in a new vlan next to our rdu3 stuff)

This affects:

vmhost-x86-cc01.rdu-cc.fedoraproject.org:pagure-stg01.fedoraproject.org:running:1
vmhost-x86-cc01.rdu-cc.fedoraproject.org:pagure02.fedoraproject.org:running:1
vmhost-x86-cc01.rdu-cc.fedoraproject.org:proxy03.fedoraproject.org:running:1
vmhost-x86-cc01.rdu-cc.fedoraproject.org:smtp-auth-cc-rdu01.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:download-cc-rdu01.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:noc-cc01.rdu-cc.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:proxy14.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:smtp-mm-cc-rdu01.fedoraproject.org:running:1
vmhost-x86-cc06.rdu-cc.fedoraproject.org
vmhost-x86-cc05.rdu-cc.fedoraproject.org
storinator01
vmhost-x86-copr01.rdu-cc.fedoraproject.org
vmhost-x86-copr02.rdu-cc.fedoraproject.org
vmhost-x86-copr03.rdu-cc.fedoraproject.org
vmhost-x86-copr04.rdu-cc.fedoraproject.org
vmhost-p08-copr01.rdu-cc.fedoraproject.org
vmhost-p08-copr02.rdu-cc.fedoraproject.org
vmhost-p09-copr01.rdu-cc.fedoraproject.org

Exact scheduling is stil being determined, but looking like the move will happen in novenber.

These two old hosts will be replaced by a new server:
vmhost-x86-cc06.rdu-cc.fedoraproject.org
vmhost-x86-cc05.rdu-cc.fedoraproject.org


Issue tagged with: sprint-0

2 months ago

I'm writing up acls for the new rdu3 network these are going to, and I need some input from the copr team. ;)

@praiskup @frostyx

My long ago understanding of how copr hypervisors are used was that you ssh to the hypervisor with it's ipv4 address, then start / provision the build vm and then that vm comes up with a ipv6 address and you connect to it from backend and manage it via ipv6. Is that still the case? How do you assign the ipv6 addresses? Is there a pool or ?

For backups to the storinator, do you just ssh directly to it via ipv4 ? or some other path?

Once I get these acls to the networking folks and they get them setup, I can get those 3 power9's up and we can use them to test with... they are already there, just need reinstalled and setup on the right vlan.

Is that still the case?

I think so, yes

How do you assign the ipv6 addresses? Is there a pool or ?

There is some range of reserved IP addresses that Fedora Infra told us they won't use for anything else.

I think this is our IPv6 prefix 2620:52:3:1:dead:beef:cafe:c, see https://pagure.io/fedora-infra/ansible/blob/fdfb14225bddc282fb182e57c08c51fcaa72b31e/f/roles/copr/backend/templates/provision/libvirt-new#_394

For backups to the storinator, do you just ssh directly to it via ipv4 ? or some other path?

We use rsnapshot over SSH. The relevant configuration seems to be

rsnapshot_push:
  server_host: storinator01.rdu-cc.fedoraproject.org
  backup_dir: /srv/nfs/copr-be
  cases:
    copr-be-copr-user:
      user: copr
      rsync_args: -H --relative /var/lib/copr/public_html --info=progress2
      command: rsnapshot_copr_backend
      timing_plan: copr_be
      # copr-backend's copr/resalloc pubkey
      ssh_pub_key: ...

Thanks for the info!

Status update:

'fedora-isolated' network has been setup.
I sent an initial set of network acls and such to networking and they have deployed them.
I'm moving forward with trying to bring the 3 power9's online there.

Those machines should allow us to see if things are setup as needed/working.

After that, we are expecting a new machine to replace several old rdu2-cc ones, once that arrives we will get it all provisioned.
Then, we hope to migrate pagure.io at least over to it and then the actual move of hardware/outage will happen.

Currently we are looking at early december for the hardware move.

I mostly have the new copr power9's setup. Status update in the upstream issue: https://github.com/fedora-copr/copr/issues/3786#issuecomment-3403103822

Almost done with those. Then need to wait for new machine to help with the move.

Log in to comment on this ticket.

Metadata
Boards 1
sprint-0 Status: Backlog