#2571 PHX2 Network outage - 2011-01-25 02:00 UTC

Created 5 years ago by smooge

= phenomenon =

We are moving from older netapp to newer one due to hardware limitations and aging hardware. Due to change in disk architectures and such, we will be having a total outage of Fedora services.

= reason =

= recommendation =

Outage: PHX2 Network outage - 2011-01-25 02:00 UTC

There will be an outage starting at 2011-01-25 02:00 UTC, which will last approximately 3 hours.

To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:

date -d '2011-01-25 02:00 UTC'

Reason for outage:

Moving from

Affected Services:

BFO - http://boot.fedoraproject.org/ [[br]] Bodhi - https://admin.fedoraproject.org/updates/ [[br]] Buildsystem - http://koji.fedoraproject.org/ [[br]] CVS / Source Control[[br]] DNS - ns1.fedoraproject.org, ns2.fedoraproject.org [[br]] Docs - http://docs.fedoraproject.org/ [[br]] Email system[[br]] Fedora Account System - https://admin.fedoraproject.org/accounts/ [[br]] Fedora Community - https://admin.fedoraproject.org/community/ [[br]] Fedora Hosted - https://fedorahosted.org/ [[br]] Fedora People - http://fedorapeople.org/ [[br]] Fedora Talk - http://talk.fedoraproject.org/ [[br]] Main Website - http://fedoraproject.org/ [[br]] Mirror List - https://mirrors.fedoraproject.org/ [[br]] Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ [[br]] Package Database - https://admin.fedoraproject.org/pkgdb/ [[br]] Smolt - http://smolts.org/ [[br]] Spins - http://spins.fedoraproject.org/ [[br]] Start - http://start.fedoraproject.org/ [[br]] Torrent - http://torrent.fedoraproject.org/ [[br]] Translation Services - http://translate.fedoraproject.org/ [[br]] Wiki - http://fedoraproject.org/wiki/ [[br]]

Unaffected Services:

NONE

Contact Information:

Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.

DNS will not be affected.

BFO may be affected partially.

The outage will be for 3 hours. fixing in ticket

Systems were updated to 5.6 but only ones using nfs/iscsi were rebooted

Found out that nfsv4 on systems was touch and go. Some systems were able to mount without problems and others were getting idmapd issues that could not be resolved. Decided to deal with those later as nfsv3 worked fine.

Most systems were able to mount the new iscsi without issues however the build servers had problems with not keeping a persistant cache. This caused them not to be able to bring up systems after reboot. Need to rearchitect and figure out.

Some remaining issues: {{{ weird denial of obsolete secondary01 mounts on puppet01 nfs4 issues xen07, bxen03, bxen04 requiring manual vgchange -ay }}}

Login to comment on this ticket.