#1845 The Move

Created 6 years ago by mmcgrath

= What =

The Fedora Project is moving out of our primary PHX datacenter into a new center. This will involve moving several machines, disk trays, etc, on to a truck, traveling a couple of cities over, and unloading and re-racking them.

= The problems =

PHX is setup as Fedora's central hub. It contains our primary data layer and build system. These systems will be completely unavailable for multiple days while we move things.

= Timing = Starting on December 12th we will start powering hosts down. We're expecting to bring services back online starting on the 13th but may not be completely done until December 15th.

= Services that will be down =

  • Buildsystem
  • CVS
  • Wiki
  • MirrorManager
  • translate.fedoraproject.org
  • Mail lists
  • Primary mirrors at download.fedora.redhat.com
  • Infra repo - http://infrastructure.fedoraproject.org/
  • All support systems (zabbix, cacti, puppet, etc)

= Services that will remain up = * fedoraproject.org/ (non wiki) * mirrors.fedoraproject.org * fedorapeople.org * DNS * fedorahosted.org ** Authentication via trac may be down. * fedoratalk - http://talk.fedoraproject.org/ * serverbeach1.fedoraproject.org * Gobby * Accounts - https://admin.fedoraproject.org/accounts/ * pkgdb - https://admin.fedoraproject.org/pkgdb/ * bodhi - https://admin.fedoraproject.org/updates/ * voting - https://admin.fedoraproject.org/voting/ * fedoracommunity - https://admin.fedoraproject.org/community/ * Smolt - http://smolts.org/ * @fedoraproject.org mail

= Services still in question = * zodbot (our IRC bot)

= Oddities = While some services above will be listed as available, those services may have stale data in them. Mirrormanager for example will stop testing mirrors for readiness meaning more stale mirrors will show up then normal.

Please make sure that the voting system remains online the whole time because the move is scheduled during the [https://fedoraproject.org/wiki/Elections#Committee_Elections_Schedule election period of FESCo, FAMSCo and the board (December 8-15)]. If the voting system is not up, we need to reschedule the elections.

I met with Mike McGrath, Nigel Jones, and John Rose about the election impact. We are going to open up elections earlier than planned, on December 5. In addition, if there is an unexpected, substantial outage of 8 hours or more, we will extend the end of elections by a day. For any additional day of outage, we will extend the end of elections by an additional day as well.

In addition, there will be a function available in the voting application that will allow a user to verify their already-recorded vote. While it's unlikely that an outage will occur, and it's also unlikely any outage would ruin a recorded vote, this will add a measure of confidence and security without causing us to run too far past our "30 days after release" guideline unnecessarily.

Mike, Do you know approximately what time the outage will start? I'd like to give the Docs folks a drop-dead time to push any update to the CVS so those documents will be updated on docs.fp.o.

Thanks!

We got network connectivity today in PHX2, Nigel is working on getting the db servers up, I've got the new bastion up (for mail and vpn).

Tomorrow we'll be scheduling an outage to do the cutover for the databases. We'll also be using this time to start using the new VPN. I'm sending out email notifications now.

I just moved some services into the known up and known down category. Mail's the last big one left unknown, it's being worked on but is not complete at this time.

email is flowing now on bastion3. I setup a new transport_maps in our main.cf that only has 2 entries. any mail bound for @.redhat.com or @redhat.com gets passed to ext-mx.corp.redhat.com

The rest of the mail is delivered directly. This is an improvement over our former situation since mail going to non-redhat.com addresses won't have to wait for an extra hop or 2 inside @redhat.com for delivery.

Mail delivery will be switched over soon, I'll put the transport_maps change in puppet so our system has the change as well.

and the internal hops are over to bastion3, now. postfix needs to continue to be up on bastion2 for a while until we migrate everything over to phx2

This is technically done. See https://fedorahosted.org/fedora-infrastructure/ticket/1884 for more information about tomorrows outage.

Login to comment on this ticket.