#5931 Moving staging to new ip space
Closed 2 years ago Opened 3 years ago by smooge.

Currently staging hosts are intermingled with production hosts on the 10.5.126 network. This causes a mess of firewall rules which would be simplified by moving the stg hosts to their own network. The 10.5.128 network is part of the same vlan so hosts that are currently a stg instance can have their ip address changed in DNS and then ansible and then rebuilt/reboot/nmtui as needed.


Metadata Update from @smooge:
- Issue assigned to smooge

3 years ago

Metadata Update from @smooge:
- Issue assigned to smooge

3 years ago

Metadata Update from @smooge:
- Issue tagged with: easyfix

3 years ago

Things that can be done as an easyfix:
1. make a pull request of the dns/master/phx2.fedoraproject.org file with the ips moved from 10.5.126 -> 10.5.128 numbers. It would be useful if they were 1:1 .
a. make a stg.phx2.fedoraproject.org file in the form of phx2.fedoraproject.org
b. take the lines out of phx2.fedoraproject.org -> stg.phx2.fedoraproject.org
c. put in a secondname for each of those hosts but with an ip address in the 10.5.128 namespace
noc01.stg.phx2.fedoraproject.org -> 10.5.126.2
noc01-a.stg.phx2.fedoraproject.org -> 10.5.128.41 [ matches noc01 on 10.5.126]
2. go through ansible and make a pull request for changes of ip addresses to match the new ip/space in 10.5.128 space
3. work with admin on rebuilds of boxes from 10.5.126 -> 10.5.128 space
4. remove all 10.5.126 stg names and rename all 10.5.128 -a ones to regular name.

Other solutions are possible.. please offer some.

Metadata Update from @smooge:
- Assignee reset

3 years ago

I'm taking a look at this.
I can manage point 1 by myself but i need help for point 2 and 3....considering also a nice Apprentice Workday for this

ps: nfs staging hosts should stay on 10.5.127 ? (bodhi-backend01-nfs.stg , pkgs01-nfs.stg , pkgs02-nfs.stg , mm-backend01-nfs.stg , wiki01-nfs.stg)

We can avoid the process for hosts with staging on 10.5.126 and prod on 10.5.125 ?

So the layout of networks via VLANs are
-- 10.5.126.0/24 10.5.128.0/24 bastion vlan
-- 10.5.127.0/24 NFS mount network -- it does not change
-- 10.5.125.0/24 Build network (firewalled off)
-- 10.5.129.128/25 10.5.129.131.0/24 QA network
-- 10.5.130.0/24 is mgmt interface network

Currently the bastion vlan can talk to the internet via firewall rules and is a combination of staging and production hosts. In the past it was just one network the 10.5.126 but we had the 128 added to it recently. By moving the staging hosts over from the 10.5.126 to 10.5.128 we can more easily firewall and replicate various infrastructure as needed.

Working on 1 is a good first step and the other ones would be stages where it does require a higher level of coordination between other people.

ok, point 1 almost done, just few things to clarify:

staging hosts without prod should stay on the 10.5.126.0/24 ? ( for example: gallery01.stg , loopabull01.stg , etc)

staging hosts with prod on the 10.5.125 can stay on 126 or has to be moved to 128 too? (for example: autosign01.stg 10.5.126.22 and autosign01 10.5.125.22)

db02.stg has to become 10.5.128.72 even if the prod host is commented out like ;db02 in the dns/master/phx2.fedoraproject.org ?

is not clear if darkserver-web01.stg and darkserver-web02.stg are staging hosts for darkserver01 and darkserver02 or not

Thanks

Good question as I wasn't clear.

  1. All stg hosts on the .126. need to go over to the 128 network even if they do not have a production ip address.
  2. Due to firewall changes.. I think we should keep .125 stg hosts where they are.
  3. If it is commented out it should still be moved in case it comes back.
  4. The darkserver-web hosts are staging hosts for a new class of systems. When that version of darkserver is completed there will be darkserver-web in production.

All stg hosts on the .126. need to go over to the 128 network even if they do not have a production ip address.

OK

Due to firewall changes.. I think we should keep .125 stg hosts where they are.

No staging host on .125 but only some production hosts, i'll move all staging hosts to 128 even the one with prod on 125 if is ok

If it is commented out it should still be moved in case it comes back.

the prod one is commented out, the stage no, i'll move the stage on 128 anyway if is ok

The darkserver-web hosts are staging hosts for a new class of systems. When that version of darkserver is completed there will be darkserver-web in production.

Ok, clear.

if everything is ok, i'll attach the stg.phx2.fedoraproject.org file asap

@smooge @giardia should this go to the mailing list so it can be implemented?

Thanks for the work on this. I took the ideas you had and put them together for a first step today. There is now a stg.phx2.fedoraproject.org and a 128.5.10.in-arpa file which some data in it.

Could you look at the files and see if my changes make sense to you.

is it possible to use "automatic PTR record synchronization" to populate PTR records in 128.5.10.in-arpa file or is done using some script?

Jumping on to help with this.
Per @smooge:

Mapping old STG IPs:
- 10.5.126.x map to 10.5.128.1-126
- 10.5.125.x map to 10.5.128.-129-254
- First available host address is .21 currently.

  • Removed additional that were not in that second list (like packages03)
  • I see there is a proxy01 and a proxy10 - is this a typo?
  • Standarized spacing throughout the file

Let me know if this works so far:
5931.patch

(also forgot to update the serial)

Starting work on replacing the old IPs with the new ones in Ansible:
From the ansible directory:

find . * -type f -exec \sed -i 's/$old_ip\b/$new_ip/' {} \;

If there's a better way to do this let me know. Not sure how we're going to add the new nodes that don't already have entries.

Files patch attached.

Did not change the following nodes due to already assigning 110, 111, 112, couldn't do it in a way with sed that wouldn't overwrite changes already:

  • osbsworker-x86-64-master01 from 128.110 to 128.205
  • osbsworker-x86-64-node01 from 128.111 to 128.206
  • osbsworker-x86-64-node02 from 128.112 to 128.207

5931-files.patch

OK so my original plan didn't work too well for moving hosts. There is a multiple stages needed and other people needing to work on the files also. The steps to change things are:

  1. Log into the host and change the ip address in the config files.
  2. Change the DNS to new ip address
  3. Change the ansible gateway/ip address
  4. Push the change to ansible and dns
  5. Run the nagios playbook
  6. Run the dns playbook
  7. Reboot the system
  8. Fix any errors.

My original plan was to make sure there were extra host names for each host so we could duplicate production. However because we are pushing two networks into the .128 this isn't possible. I am now just going to import the hostnames currently in staging.

To try and cut down conflicts of dns ips getting used, we are going to assign all the current .126. staging hosts ip addresses in the .128 with a hostname starting with n- . Then the steps for moving a host will be the following:

  1. Log into the host and change the network and similar files with the new ipaddress.
  2. Edit the stg.fedoraproject.org host file with the host we are 'moving' over so that its old name starts with o- and then remove the n- from the new ip address
  3. Change the ansible gateway/ip address
  4. Push the change to ansible and dns
  5. Run the nagios playbook
  6. Run the dns playbook
  7. Reboot the system
  8. Fix any errors.

After the host has been successfully updated, remove the o- from the stg.fedoraproject.org and change the entry in the 128.5.10-in.arpa file to unused.

The stg.fedoraproject.org and 128.5.10.reverse have been updated to match the temp name scheme. All that needs to be done is fixing the ansible config files to match these ips versus the other ones. One other change that needs to be done for each inventory file is that is changed needs to have its gateway switched to 10.5.128.254 also . I would look at doing something like a

find ansible/inventory -type f -print | xargs grep -l 10.5.128 and change the gw: to match 10.5.128.254 either through another sed or similar.

Delaying until after 2017 FLOCK, ending 09/01/2017.

Diff attached. One of the requested IPs to change was for mbs-frontend02.stg.phx2.fedoraproject.org , however, I couldn't find an n-* entry for it in the zone file, so I changed it anyway to what was listed for mbs-frontend02in the zone file: 10.5.128.146

5931-host-vars.patch

Moving is complete. We 98 free host addresses on 10.5.126.x and 89 free host addresses on 10.5.128.x

This issue can be closed.

Metadata Update from @smooge:
- Issue assigned to smooge

2 years ago

Thanks to everyone who helped on this. IT IS DONE

:birthday:

Metadata Update from @smooge:
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Attachments 6
Attached 2 years ago View Comment
Attached 2 years ago View Comment
Attached 2 years ago View Comment
Attached 2 years ago View Comment
Attached 2 years ago View Comment