Problem System State

==================================================================================
Enabling system bus....... Done
Initializing CPUs......... Done
Initializing boot memory.. Done
Initializing OpenBoot
Probing system devices
Probing I/O buses
screen not found.
keyboard not found.
Keyboard not present. Using ttya for input and output.
Probing system devices
Probing I/O buses

Sun Fire 880, No Keyboard

Copyright 2007 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.22.34, 8448 MB memory installed, Serial #54838170.
Ethernet address 0:3:ba:44:c3:9a, Host ID: 8344c39a.

Rebooting with command: boot

Boot device: disk0 File and args:
SunOS Release 5.9 Version Generic_117171-10 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed
WARNING: forceload of misc/md_sp failed
configuring IPv4 interfaces: ce0 ce1.
Hostname: svr2
Core files in /var/tmp/SUNWscu/core: core.svr2.didadm.1955.1403589151
core.svr2.didadm.1955.1403590509 core.svr2.didadm.1955.1403591961
core.svr2.picld.1955.1368019720 core.svr2.picld.1965.1365064821
core.svr2.picld.1966.1365133575 core.svr2.picld.1966.1368024863
core.svr2.picld.1966.1368167875 core.svr2.picld.1966.1368524105
core.svr2.picld.1966.1368678586 core.svr2.picld.1966.1369034988
core.svr2.picld.1966.1369655020 core.svr2.picld.1966.1369655949
core.svr2.picld.1966.1370756790 core.svr2.picld.1968.1368529734 core.svr2.picld.854.1365242684
device id for '/dev/rdsk/c1t3d0' does not match physical disk's id.
The drive may have been replaced
device id for '/dev/rdsk/c1t1d0' does not match physical disk's id.
The drive may have been replaced
device id for '/dev/rdsk/c1t2d0' does not match physical disk's id.
The drive may have been replaced
device id for '/dev/rdsk/c1t0d0' does not match physical disk's id.
The drive may have been replaced
Could not open /dev/rdsk/c1t4d0s2 to verify device id.

No such device or address

device id for '/dev/rdsk/c1t5d0' does not match physical disk's id.
The drive may have been replaced
device id for '/dev/rdsk/c2t0d2' does not match physical disk's id.
The drive may have been replaced
Booting as part of a cluster
NOTICE: CMM: Node svr2 (nodeid = 1) with votecount = 1 added.
NOTICE: CMM: Node svr1 (nodeid = 2) with votecount = 0 added.
NOTICE: clcomm: Adapter qfe0 constructed
NOTICE: clcomm: Path svr1:qfe0 - svr2:qfe0 being constructed
NOTICE: clcomm: Adapter eri0 constructed
NOTICE: clcomm: Path svr1:eri0 - svr2:eri0 being constructed
NOTICE: CMM: Node svr1: attempting to join cluster.
NOTICE: CMM: Cluster doesn't have operational quorum yet; waiting for quorum.
NOTICE: clcomm: Path svr1:qfe0 - svr2:qfe0 errors during initiation
WARNING: Path svr1:qfe0 - svr2:qfe0 initiation encountered errors, errno = 62. Remote node may be
down or unreachable through this path.
NOTICE: clcomm: Path svr1:eri0 - svr2:eri0 errors during initiation
WARNING: Path svr1:eri0 - svr2:eri0 initiation encountered errors, errno = 62. Remote node may be
down or unreachable through this path.
=================================================================================

Problem Summary
Operational node svr1 gone down due to unknown failure which was owning the cluster quorum
device which caused the node svr2 not able to boot in cluster mode and struck at the above
mentioned state.
=================================================================================

Workaround for the Problem

Boot svr2 in non cluster mode
ok boot -x
# cd /etc/cluster/ccr
# /usr/bin/cp infrastructure infrastructure.old

Get this node's id.

# cat /etc/cluster/nodeid
2

Edit the /etc/cluster/ccr/infrastructure file

cluster.nodes.2.name
svr2
cluster.nodes.2.state
enabled
cluster.nodes.2.properties.quorum_vote

 For all other node and any Quorum Device, set the vote count to zero. Other nodes, where is
any node id but the one edited above:
cluster.nodes.1.name
svr1
cluster.nodes.1.state
enabled
cluster.nodes.1.properties.quorum_vote

1 =====> change it to 0

Quorum Device(s)

cluster.quorum_devices.1.properties.votecount

1 ======> change it to 0

Regenerate the checksum of the infrastructure file by running:

# /usr/cluster/lib/sc/ccradm -i /etc/cluster/ccr/infrastructure -o