1/9/2021 Document Display

Copyright (c) 2021, Oracle. All rights reserved. Oracle Confidential.

Solaris Cluster Reservation Conflict Panic in a Clustered Guest LDom When Rebooting
Primary/Service Domain (Doc ID 1263669.1)

APPLIES TO:

Solaris Cluster - Version 3.2 and later

Oracle Solaris on SPARC (32-bit)
Oracle Solaris on SPARC (64-bit)

SYMPTOMS

You have a Solaris Cluster configured where one of the node is a guest logical domain.
And that domain panicked with the following stack trace:

Oct 1 08:38:11 node1
node is panicking
Oct 1 08:38:11 node1
Oct 1 08:38:11 node1
Oct 1 08:38:11 node1
Oct 1 08:38:11 node1
cl_dlpitrans: [ID 624622 kern.notice] Notifying cluster that this
unix: [ID 836849 kern.notice]
panic[cpu30]/thread=2a104867ca0:
unix: [ID 632503 kern.notice] Reservation Conflict
Disk: /virtual-devices@100/channel-devices@200/disk@2

unix:panicsys+0x48()
unix:vpanic_common+0x78()
unix:panic+0x1c()
vdc:vdc_scsi_status+0x29c()
vdc:vdc_failfast_scsi_cmd+0x94()
vdc:vdc_failfast_check_resv+0x24()
vdc:vdc_failfast_thread+0x38()
unix:thread_start+0x4()
-- end of kernel thread's stack --

CAUSE

In a clustered guest logical domain, access to the private network interfaces, system disk, shared and non-shared disks is
provided by virtual I/O devices.
These virtual devices are being serviced by a service domain. The service domain has direct I/O access to the physical
devices and presents these devices
as virtual devices to guest domains.

Let's take an example on how the devices are configured.

Not all network devices are shown here.

Network devices:
For the primary/service domain:

VSW NAME MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE
.....
vsw2 00:14:4f:fa:39:c0 nxge10 2 switch@2 phys-state 1 1 1500 sc
vsw6 00:14:4f:f9:da:c4 nxge14 6 switch@6 phys-state 1 1 1500 sc

For the guest domain:

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=qnrpgf94q_4 1/4
1/9/2021 Document Display

NAME SERVICE ID DEVICE MAC MODE PVID VID MTU LINKPROP

vnet4 vsw2@primary 4 network@4 00:14:4f:f8:d8:b9 1 1500 phys-state
vnet5 vsw6@primary 5 network@5 00:14:4f:f8:0d:cd 1 1500 phys-state

Disk devices:
For the primary/service domain:

VDS
NAME VOLUME OPTIONS MPGROUP DEVICE
primary-vds0 c1t2d0 /dev/dsk/c1t2d0s2
c1t3d0 /dev/dsk/c1t3d0s2
vol_01 /dev/dsk/c6t60080E500017E502000002EE4C4CB02Fd0s2
vol_03 /dev/dsk/c6t60080E500017E502000002E24C4CA66Ed0s2
s10_dvd1 ro /export/iso/sol-10-u8-ga-sparc-dvd.iso

For the guest domain:

DISK
NAME VOLUME TOUT ID DEVICE SERVER MPGROUP
vdisk0 c1t2d0@primary-vds0 600 0 disk@0 primary
vdisk1 c1t3d0@primary-vds0 600 1 disk@1 primary
vshare0 vol_01@primary-vds0 600 2 disk@2 primary
vshare1 vol_03@primary-vds0 600 3 disk@3 primary
s10_dvd s10_dvd1@primary-vds0 600 4 disk@4 primary

The node1 is the name of the system installed in the guest domain.
It is clustered with a system named node2.

A couple of minutes before the panic, the following messages are reported on node1.
It indicates that all virtual disk and virtual network devices are offline/down on the guest logical domain.

Oct 1 08:36:32 node1 vdc: [ID 990228 kern.info] vdisk@1 is offline

Oct 1 08:36:32 node1 vdc: [ID 990228 kern.info] vdisk@4 is offline
Oct 1 08:36:32 node1 vdc: [ID 990228 kern.info] vdisk@0 is offline
Oct 1 08:36:32 node1 vdc: [ID 990228 kern.info] vdisk@2 is offline
Oct 1 08:36:32 node1 vdc: [ID 990228 kern.info] vdisk@3 is offline
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet0: link down
Oct 1 08:36:34 node1 in.mpathd[242]: [ID 215189 daemon.error] The link has gone down
on vnet0
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet1: link down
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet2: link down
Oct 1 08:36:34 node1 Cluster.PNM: [ID 890413 daemon.notice] ipmp0: state transition
from OK to DOWN.
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet3: link down
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet4: link down
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet5: link down
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet6: link down
Oct 1 08:36:34 node1 vnet: [ID 134599 kern.info] vnet7: link down

So the guest logical domain cannot access anymore any of the disks or network devices.
But the Solaris operating system still runs in the kernel and this cluster node tries to send/receive heartbeats to the other
cluster node.

However since the network devices are down, the node will not be able to send/receive any heartbeats.
After a couple of seconds (10 by default), the cluster will declare the private links down and enter a split brain situation.

Oct 1 08:36:42 node1 cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path
jsws0011:vnet4 - jsws0012:vnet4 being drained

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=qnrpgf94q_4 2/4
1/9/2021 Document Display
Oct 1 08:36:42 node1 cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path
jsws0011:vnet5 - jsws0012:vnet5 being drained

Node1 will try to take ownership of the quorum device by sending an ioctl to the corresponding to the virtual disk.
But since all the virtual disks are offline, the ioctl will be blocked.

At the same time, node2 also realized that there is a split brain.
It tried and succeeded to take ownership of the quorum device.
So node2 is alone in the cluster and is fencing node1 from all shared disks.

Back to node1, messages are reported indicating that some disk devices are online again.

Oct 1 08:38:02 node1 vdc: [ID 625787 kern.info] vdisk@0 is online using ldc@8,0
Oct 1 08:38:02 node1 vdc: [ID 625787 kern.info] vdisk@1 is online using ldc@9,0
Oct 1 08:38:09 node1 vdc: [ID 625787 kern.info] vdisk@2 is online using ldc@10,0

Since the virtual disks are online, pending I/O that were blocked due to the disks being offline can resume.
And also new I/Os can be sent to these virtual disks.
But node2 has fenced off node1 from the shared disks, so any I/O to shared disks will return a reservation conflict
status.

And this is causing node1 to panic with reservation conflict as per design.

The service domain has direct I/O access to the physical devices. When it is rebooted, dependent LDOMs lose access to
these physical devices.
With nothing servicing these physical devices to the guest domain the virtual devices are going offline/down in the Guest
Domains.

It means that no heartbeats sent by the other cluster node will arrive on the service domain and then no heartbeats can
be routed to the guest domain.
Additionally, node1 cannot send any heartbeats to node2.

Once the service domain reboots, the virtual devices come online/up in the guest domain and can be serviced again.
But at that time, node1 has already been fenced off from shared disks.

The whole issue is due to virtual disks and virtual network devices going offline/down.
And this has been caused by a reboot of the service domain.

SOLUTION

This is a fully expected behavior.

If you are running Solaris Cluster in a guest domain, understand that rebooting the service domain may have an impact on
the running guest domain,
including domain going down. You should rebalance the workload to other cluster nodes first and then stop the guest
domains running Solaris Cluster
before rebooting the service domain.

Additionally, you should setup a failure-policy and Master/slave dependency as noted in the Oracle Solaris Cluster Data
Service for Oracle VM Server for SPARC Guide

Also see: Solaris Cluster Failover LDom Resource Creation Fails with "Master-slave dependency between primary domain
and guest domain does not exist." (Doc ID 2189857.1)

Note that the auto-boot? setting should be true for Guest LDOM's which are Solaris Cluster nodes (not failover ldoms).

Didn't find what you are looking for?

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=qnrpgf94q_4 3/4
1/9/2021 Document Display

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=qnrpgf94q_4 4/4