Stop-DatabaseAvailabilityGroup

Has the datacenter

switchover been approved?
Is the primary datacenter
online or physically
accessible?
Is your DAG extended to
Multiple Active Directory
Sites?
Are the Exchange servers in
primary datacenter online?
Do the remote and primary
datacenters have network
connectivity?
Are the Exchange servers in
primary datacenter online?
Is your DAG extended to
Multiple Active Directory Sites?
Is your DAG extended to
Multiple Active Directory Sites?
Stop-DatabaseAvailabilityGroup
COMMANDS:
Using the Exchange Management Shell on a sever in the recovery
datacenter, run:
Stop-DatabaseAvailabilityGroup Identity <DAGName> -MailboxServer
<DAG member in primary site> -ConfigurationOnly:$True
Repeat command for all DAG members that are not in the recovery
datacenter.
EXPECTED OUTCOMES:
1)Verify the servers on the StartedMailboxServers and
StoppedMailboxServers lists for the DAG:
Get-DatabaseAvailabilityGroup Identity <DAGName> | FL
The StoppedMailboxServer should list all mailbox servers in the primary
datacenter and the StartedMailboxServers should list all mailbox servers in
the recovery datacenter.
2)A double write to both a domain controller in the recovery datacenter and
a domain controller in the primary datacenter of the StoppedMailboxServers
attribute is performed. This is done to bypass Active Directory site
replication latency.
COMMON ERRORS:
If a domain controller in the primary datacenter is not available, the
command may return an Active Directory provider error. This error can be
safely ignored.
COMMANDS:
Using the Exchange Management Shell on a sever in the recovery
datacenter, run:
Stop-DatabaseAvailabilityGroup Identity <DAGName> -MailboxServer
<DAG member in primary site> -ConfigurationOnly:$True
Repeat command for all DAG members that are not in the recovery
datacenter.
EXPECTED OUTCOMES:
1)Verify the servers on the StartedMailboxServers and
StoppedMailboxServers lists for the DAG:
Get-DatabaseAvailabilityGroup Identity <DAGName> | FL
The StoppedMailboxServer should list all mailbox servers in the primary
datacenter and the StartedMailboxServers should list all mailbox servers in
the recovery datacenter.
2)A double write to both a domain controller in the recovery datacenter and
a domain controller in the primary datacenter of the StoppedMailboxServers
attribute is performed. This is done to bypass Active Directory site
replication latency.
COMMON ERRORS:
If a domain controller in the primary datacenter is not available, the
command may return an Active Directory provider error. This error can be
safely ignored.
COMMANDS:
Using the Exchange Management Shell on a sever in the recovery datacenter, run:
Stop-DatabaseAvailabilityGroup Identity <DAGName> -MailboxServer
<DAG member in primary site>
Repeat the above command for all DAG members that are not in the recovery
datacenter.
EXPECTED OUTCOMES:
1)Verify the servers on the StartedMailboxServers and StoppedMailboxServers
lists for the DAG:
Get-DatabaseAvailabilityGroup Identity <DAGName> | FL
The StoppedMailboxServer should list all mailbox servers in the primary
datacenter and the StartedMailboxServers should list all mailbox servers in the
recovery datacenter.
2)Exchange servers that were accessible in the primary datacenter should have
their Cluster services forcibly cleaned up and the Cluster service should be
configured with a startup type of DISABLED. You can verify this using
Services.msc.
3)A double write to both a domain controller in the recovery datacenter and a
domain controller in the primary datacenter of the StoppedMailboxServers
attribute is performed. This is done to bypass Active Directory site replication
latency.
COMMON ERRORS:
If a domain controller in the primary datacenter is not available, the command
may return an Active Directory provider error. This error can be safely ignored.
COMMANDS:
Using the Exchange Management Shell on a sever in the recovery datacenter, run:
Stop-DatabaseAvailabilityGroup Identity <DAGName> -ActiveDirectorySite
<primary site>
Repeat the above command for all Active Directory sites containing DAG members
that are not the recovery datacenter AD site.
EXPECTED OUTCOMES:
1)Verify the servers on the StartedMailboxServers and StoppedMailboxServers lists
for the DAG:
Get-DatabaseAvailabilityGroup Identity <DAGName> | FL
The StoppedMailboxServer should list all mailbox servers in the primary
datacenter and the StartedMailboxServers should list all mailbox servers in the
recovery datacenter.
2)Exchange servers that were accessible in the primary datacenter should have
their Cluster services forcibly cleaned up and the Cluster service should be
configured with a startup type of DISABLED. You can verify this using Services.msc.
3)A double write to both a domain controller in the recovery datacenter and a
domain controller in the primary datacenter of the StoppedMailboxServers
attribute is performed. This is done to bypass Active Directory site replication
latency.
COMMON ERRORS:
If a domain controller in the primary datacenter is not available, the command
may return an Active Directory provider error. This error can be safely ignored.
Start-DatabaseAvailabilityGroup
Is your primary datacenter
online?
Start-DatabaseAvailabilityGroup
Ensure that supporting services are available including but not limited to:
1) Active Directory / domain controllers / global catalog / FSMO role
holders
2) Domain Name Services (DNS)
3) Witness Server
4) Supporting Exchange roles: Client Access and Hub Transport
OPTIONAL:
Dynamic Host Configuration Protocol servers (DHCP), if DHCP addresses are
used for DAG networks
Edge Transport server
Unified Messaging server
Are the necessary services
established and functioning?
COMMANDS:
Verify network connectivity between all DAG members.
Suggested methods:
1) Ping test between DAG members
2) Map administrative shares between DAG members
EXPECTED OUTCOMES:
Connectivity between datacenters is functioning and all cluster
inter-node communications are operating normally
Have datacenter
communications been
verified?
Verify that Cluster service on the DAG members in
the primary datacenter have a startup type of
DISABLED. If they do not, either the Stop-
DatabaseAvailabilityGroup command was not
successful or the DAG members in the primary
datacenter failed to receive eviction notification
after network connectivity between datacenters
was restored
Do not proceed until Cluster service cleanup has
occurred and Cluster service has a startup type
of DISABLED.
You can optionally run the following command on
the DAG members in the primary datacenter to
forcibly cleanup the outdated cluster information:
Cluster node /forcecleanup
Does the Cluster service
show a startup type of
disabled?
Is your DAG extended to
multiple Active Directory
sites?
COMMAND:
Using the Exchange Management Shell, run the following command:
Start-DatabaseAvailabilityGroup Identity <DAGName> -ActiveDirectorySite <primary site>
Repeat for all other Active Directory sites that were stopped during the datacenter switchover process.
EXPECTED OUTCOMES:
1) DAG members in the primary datacenter are added to the DAGs cluster
2) If the resulting membership count is EVEN, the cluster is to use the Node and File Share Majority
quorum
VERIFICATION:
Use the following steps to verify that the DAG members are up and the Cluster Group is online by running the
following commands:
Windows Server 2008 R2
1) Import-Module FailoverClusters
2) Get-ClusterNode Cluster <DAGName>
3) Get-ClusterGroup Cluster <DAGName>
Windows Server 2008 SP2
1) Cluster <DAGName> node
2) Cluster <DAGName> group
The following command shows the StartedMailboxServers list with all DAG members and an empty
StoppedMailboxServers list:
Get-DatabaseAvailabilityGroup Identity <DAGName> | FL
COMMON ERRORS:
Nodes may fail to join the cluster with invalid node error. If this occurs, retry the command again.
COMMAND:
Using the Exchange Management Shell, run the following command:
Start-DatabaseAvailabilityGroup Identity <DAGName> -MailboxServer <DAG
member in primary site>
Repeat for all other Mailbox servers that were stopped during the datacenter switchover
process.
EXPECTED OUTCOMES:
1) DAG members in the primary datacenter are added to the DAGs cluster
2) If the resulting membership count is EVEN, the cluster is to use the Node and File
Share Majority quorum
VERIFICATION:
Use the following steps to verify that the DAG members are up and the Cluster Group is
online by running the following commands:
Windows Server 2008 R2
1) Import-Module FailoverClusters
2) Get-ClusterNode Cluster <DAGName>
3) Get-ClusterGroup Cluster <DAGName>
Windows Server 2008 SP2
1) Cluster <DAGName> node
2) Cluster <DAGName> group
The following command shows the StartedMailboxServers list with all DAG members and an
empty StoppedMailboxServers list:
Get-DatabaseAvailabilityGroup Identity <DAGName> | FL
COMMON ERRORS:
Nodes may fail to join the cluster with invalid node error. If this occurs, retry the command
again.
Were the DAG Members added
to the cluster Successfully?
COMMANDS:
Reset the DAGs Witness Server and Alternate Witness Server properties by running the following command:
Set-DatabaseAvailabilityGroup Identity <DAGName> -WitnessServer <WSName> -AlternateWitnessServer <AWSName>
EXPECTED OUTCOMES:
Witness Server and Alternate Witness Server properties are configured to ensure the appropriate witness server is in use
If the Cluster configuration does not match the DAG configuration, the Cluster is updated with the proper configuration
COMMON ERRORS:
Administrators incorrectly verify which file share witness is currently in use. See http://aka.ms/E14FSW.
After any activation blocks have been removed,
active database copies can be moved to servers in
the primary datacenter
22 October 2012
Were the DAG Members added
to the cluster Successfully?
NO
YES
Command Completed
NO
YES
Command Completed
YES
YES YES
YES YES
Y
E
S
YES
Y
E
S
YES
YES
YES
YES
Y
E
S
Y
E
S
NO
N
O
NO NO
N
O
NO
NO
NO
N
O
N
O
NO
NO
N
O
N
O
N
O
N
O
N
O
NO
N
O
N
O
C
o
m
p
le
te
d
Completed
C
o
m
p
le
te
d
Restore-DatabaseAvailabilityGroup
Is the Cluster service stopped on all
DAG members in your recovery
datacenter?
COMMANDS:
Stop the Cluster service on each DAG member in the
recovery datacenter. To do this run the appropriate
command for your DAG members operating system:
Windows Server 2008 R2: Stop-Service Clussvc
Windows Server 2008 SP2: Net Stop Clussvc
EXPECTED OUTCOMES:
Cluster services are stopped on remaining nodes.
COMMON ERRORS
Access denied You must use an elevated command
prompt run as administrator if the default administrator
account is not used
COMMANDS:
From the Exchange Management Shell on an Exchange server in the recovery datacenter, run:
Restore-DatabaseAvailabilityGroup Identity <DAGName> -ActiveDirectorySite <recovery site>
-AlternateWitnessDirectory:<AWSPath> -AlternateWitnessServer:<AWSName>
EXPECTED OUTCOMES:
1) A DAG member in the recovery datacenter is randomly selected and its Cluster service is started in /
forceQuourm mode
2) DAG members on the StoppedMailboxServers list are evicted from the DAGs cluster thereby adjusting
the membership count
a) If the resulting membership count is EVEN or results in a SINGLE node, the Cluster is configured with a
Node and File Share Majority quorum and it begins using the Alternate Witness Server and Alternate Witness
Directory
3) Cluster services are started on the remaining DAG members and they successfully join the DAGs
cluster
VERIFICATION:
Use the following steps to verify that the DAG members are up and the Cluster Group is online by running the
following commands:
Windows Server 2008 R2
1) Import-Module FailoverClusters
2) Get-ClusterNode Cluster <DAGName>
3) Get-ClusterGroup Cluster <DAGName>
Windows Server 2008 SP2
1) Cluster <DAGName> node
2) Cluster <DAGName> group
COMMON ERRORS:
Nodes fail to evict with error 0x46. See http://aka.ms/0x46
Restore-DatabaseAvailabilityGroup
Assuming all pre-requisites have
been met, any activation blocks can
now be removed and databases can
be mounted
NO
YES
Command Completed
Y
E
S
Y
E
S
N
O
C
o
m
p
le
te
d

Exchange 2010 Datacenter Switchover
Troubleshooter Diagram
LEARN MORE
http://technet.microsoft.com/en-us/library/dd351049.aspx
Diagram: Serkan Varoglu
Exchange 2010 includes a feature called Datacenter Activation Coordination (DAC) mode that is designed to prevent split brain at the database level during
switchback procedures that are being performed after a datacenter switchover has occurred. One of the side benefits of enabling DAC mode is that it
enables you to use the built-in recovery cmdlets to perform the datacenter switchover and switchback.
This diagram allows administrators to work through the flow of questions to determine:
What commands to run and where to run them
How to verify the commands completed successfully.
How to walk through a Datacenter Switchover from the Mailbox server / database availability group perspective.
This diagram is created as an alternate to Exchange 2010 Datacenter Switchover Troubleshooter (Powerpoint tool) authored by TimMcMichael.
If you want to use the original tool please go to:
http://blogs.technet.com/b/exchange/archive/2012/10/19/exchange-2010-datacenter-switchover-troubleshooter-now-available.aspx
@TheUCArchitects
TheUCArchitects.com @SRKNVRGL
Diagram v1.0 Troubleshooter v1.1