LCM 2.

Life Cycle Manager

Dark Site Guide
September 3, 2020
Contents

1.  Overview...................................................................................................................... 4
The Life Cycle Manager Without Internet Access........................................................................................4
Prism Central and Prism Element....................................................................................................................... 5

2.  Opening LCM............................................................................................................ 6

Accessing the Life Cycle Manager from Prism Element............................................................................6
Accessing the Life Cycle Manager from Prism Central.............................................................................. 7

3. Setting Up a Local Web Server........................................................................ 9

Setting Up a Local Web Server (Windows)................................................................................................... 9
Setting Up a Local Web Server (Linux).......................................................................................................... 11

4. Updating the LCM Framework......................................................................... 13

Updating the LCM Framework at a Dark Site.............................................................................................. 13

5. LCM Software Updates........................................................................................15

Fetching the AHV Dark Site Bundle................................................................................................................ 15
Fetching the Foundation Dark Site Bundle...................................................................................................16
Fetching the Calm Dark Site Bundle............................................................................................................... 16
Updating Software at a Dark Site..................................................................................................................... 17

6. LCM Firmware Updates..................................................................................... 20

Dark Site Firmware Updates.............................................................................................................................. 20
Effects of Updates on the Cluster................................................................................................................... 23

7.  LCM Prechecks....................................................................................................... 26

Life Cycle Manager Pre-Checks........................................................................................................................ 26

8.  LCM Log Collection............................................................................................. 29

Collecting Life Cycle Manager Logs................................................................................................................29

9.  LCM Glossary......................................................................................................... 30

Glossary.......................................................................................................................................................................30

Copyright...................................................................................................................32
License......................................................................................................................................................................... 32

ii
Conventions............................................................................................................................................................... 32
Default Cluster Credentials..................................................................................................................................32
Version......................................................................................................................................................................... 33

iii
1
OVERVIEW
The Life Cycle Manager Without Internet Access
Use the Life Cycle Manager at a dark site.

LCM
The Life Cycle Manager (LCM) tracks software and firmware versions of all entities in the
cluster.
This document assumes that you are using LCM at a dark site, a location without Internet
access. To use LCM at a connected site, see the Life Cycle Manager Guide.

LCM Structure
LCM consists of a framework and a set of modules for inventory and update.
LCM supports software updates for all platforms that use Nutanix software.
LCM supports firmware updates only for the following platforms.

Table 1: Platforms That Support Firmware Updates Through LCM

Nutanix (NX) Dell XC / XC Core Lenovo HX / HX Ready

HPE DX Fujitsu XF Intel DCB

HPE DL (G10) Inspur InMerge

Access the LCM framework through the Prism interface. Prism acts as a download manager for
LCM modules, validating and downloading module content. All communication between the
cluster and LCM modules goes through the LCM framework.
LCM modules are independent of AOS. They contain libraries and images, as well as metadata
and checksums for security. Currently Nutanix supplies all modules.

LCM Support
LCM supports both Prism Element and Prism Central.
Nutanix recommends that you use LCM to update your cluster to the most recent version of
Nutanix Foundation before performing any other LCM updates.
Nutanix recommends that you perform all updates through LCM, where available. However,
your platform vendor may recommend updates that are not yet available through LCM. The lag
is caused by the time it takes Nutanix to revalidate the LCM payload after incorporating new
updates from vendors.

LCM |  Overview | 4
 LCM Operation
LCM performs two functions: taking inventory of the cluster and performing updates on the
cluster. LCM updates are not reversible.
Before performing an update, LCM runs a set of pre-checks to verify the state of the cluster. If
any checks fail, LCM stops the update.
LCM writes all operations to output logs:

• genesis.out

• lcm_ops.out

• lcm_ops.trace

• lcm_wget.log
The log files record all operations, including successes and failures. If an operation fails, LCM
suspends it to wait for mitigation. Contact Nutanix Support for assistance if there is an LCM
failure.
The LCM framework can also update itself when necessary. Although connected to AOS, the
framework is not part of the AOS release cycle.

Limitations

• LCM cannot perform firmware upgrades on single-node clusters. Firmware updates usually
require services to stop, and on a single-node cluster there is no other node to take over the
workload. This limitation does not apply to software updates.
• LCM is a separate process from Prism one-click upgrades. CLI commands specific to the
one-click upgrade workflow, such as firmware_upgrade_status, do not apply to LCM. For
update status, use lcm_update_status.

• LCM cannot operate when you have VLAN traffic filtering (such as VLAN trunking or virtual
guest tagging) deployed on the dvPortGroup connected to the Controller VM eth0 interface.

Prism Central and Prism Element

Information on using LCM with Prism Central and Prism Element.

Prism Central
Some entities are local to Prism Central, such as Calm, Epsilon, Karbon, and Objects. To manage
these entities, you must log on to Prism Central.
Other entities, such as component firmware, are local to Prism Element. To manage these
entities, you must log on to Prism Element.
In general, follow these guidelines for using LCM with Prism Element and Prism Central:

• Make sure that Prism Central and Prism Element use the same LCM URL. (At a dark site,
make sure that both Prism Central and Prism Element use the same dark site bundle.)
• Before you access Prism Element from Prism Central, perform an LCM inventory on Prism
Central, using cluster quick access.
• Every time you register a new Prism Element instance to Prism Central, perform an LCM
inventory on Prism Central.
• Nutanix recommends that you enable auto-inventory for LCM on Prism Central, to make sure
that the UI stays up to date on all Prism Element instances.

LCM |  Overview | 5
2
OPENING LCM
Accessing the Life Cycle Manager from Prism Element
Open LCM from Prism Element.

About this task

Different versions of AOS use different methods of opening entities in Prism.

Procedure
Open LCM with the method for your version of AOS.

» For AOS 5.11 and later:

In Prism, open the drop-down menu on the upper left and select LCM.

LCM |  Opening LCM | 6

 » For AOS 5.10:
In Prism, click the gearbox button to open the Settings page and select Life Cycle
Management in the sidebar.

» For AOS 5.5:

In Prism, click the gearbox button and select Life Cycle Management from the drop-down
menu.

Accessing the Life Cycle Manager from Prism Central

Open LCM from Prism Central.

About this task

Different versions of AOS use different methods of opening entities in Prism.

Procedure
Open LCM with the method for your version of AOS.

LCM |  Opening LCM | 7

» For AOS 5.11 and later:
In Prism Central, open the drop-down menu on the upper left and select Administration >
LCM.

» For AOS 5.10:

In Prism Central, click the gearbox button to open the Settings page and select Life Cycle
Management in the sidebar.
3
SETTING UP A LOCAL WEB SERVER
Setting Up a Local Web Server (Windows)
Set up a local web server for dark site updates.

About this task

To host the LCM repository, set up a local web server that is reachable by all your Nutanix
clusters. For IIS information and examples, see the Microsoft documentation.

Note: IIS only accepts directory and file names in English characters. To avoid errors, make sure
that you have Windows set to the English-language setting.

Note: If you use HTTPS, it is your responsibility to make sure that the certificates are correct.
LCM only accepts certificates issued by a trusted certificate authority (CA). Certificates from a
custom CA are not accepted.

Do not use a Nutanix Controller VM as a web server. Use a separate VM or a physical server.

Procedure

1. In IIS, create a virtual directory called release.

Figure 1: Add virtual directory

When you download your dark site bundles in the following chapters, make sure that you
extract the dark site bundles into the release directory. LCM must be able to look up the
manifest at http://web_server_IP_address/release/master_manifest.tgz.sign.

LCM |  Setting Up a Local Web Server | 9

2. Enable directory browsing.

Figure 2: Add virtual directory

3. In the release directory, create or modify the following MIME types.

a. Create a MIME type called '.sign' with the type set as text/plain.

b. Create a MIME type called '.json' with the type set as text/plain.

c. Create a MIME type called '.iso' with the type set as text/plain.

d. Create a MIME type called '.xz' with the type set as text/plain.

e. Make sure that MIME types with the following extensions exist in the release directory.
Some are present by default. For any that are not present by default, create them and set
the type as application/x-iso9660-image.

• .BD

• .bin

• .csv

• .exe

• .frm

• .lod

• .md

• .std

• .tgz

• .vib

f. IIS has request filtering disabled by default. If you have enabled request filtering, make
sure that you configure exceptions for these MIME types.

4. Set permissions on the release directory to match the following image.

Figure 3: IIS permissions

5. Make sure that the files in your web server are all in lower case. Windows is case-insensitive,
but the LCM framework ignores files in upper case.

LCM |  Setting Up a Local Web Server | 10

Setting Up a Local Web Server (Linux)
Set up a local web server for dark site updates.

Before you begin

This procedure requires the following:

• A server running CentOS 7 with an installation package of server with GUI

• A static IP address for your server
• A client to test your installation

About this task

To host the LCM repository, set up an Apache web server that is reachable by all your Nutanix
clusters.

Note: If you use HTTPS, it is your responsibility to make sure that the certificates are correct.
LCM only accepts certificates issued by a trusted certificate authority (CA). Certificates from a
custom CA are not accepted.

Do not use a Nutanix Controller VM as a web server. Use a separate VM or a physical server.

Procedure

1. Update the yum repository on your Linux server.

[root@darksitewebcentos ~]# sudo yum update -y

2. Install httpd.

[root@darksitewebcentos ~]# sudo yum install httpd -y

3. Check that SELinux is enabled.

[root@darksitewebcentos ~]# getenforce

If SELinux is enabled, the command returns the string Enforcing.

If SELinux is disabled, the command returns the string Disabled.

4. To enable or disable SELinux, use the following command.

[root@darksitewebcentos ~]# setenforce 1|0

5. Set the firewall to allow Apache access.

[root@darksitewebcentos ~]# sudo firewall-cmd --permanent --zone=public --add-service=http

LCM |  Setting Up a Local Web Server | 11

6. Reload the firewall to make your changes take effect.

[root@darksitewebcentos ~]# sudo firewall-cmd --reload

7. In a text editor, open the main server configuration file at /etc/httpd/conf/httpd.conf.

8. Make sure that httpd.conf contains the following lines. If they are commented out,
uncomment them.

ServerRoot “/etc/httpd”
DocumentRoot “/var/www/html” #All the documents will be served under this directory
hierarchy
ServerName your-web-server-IP-address|Fully qualified domain name #if nameservice is
available
Listen IP-address:port

9. Restart httpd.

[root@darksitewebcentos ~]# sudo systemctl restart httpd

10. Enable Apache to start when the system starts.

[root@darksitewebcentos ~]# sudo systemctl enable httpd

11. Open a browser and enter the static IP address assigned to your web server in the search
bar.
You get the following result page.

Figure 4: Result

12. Create a directory called release under the document root /var/www/html.

[root@darksitewebcentos ~]# mkdir -p -m 755 /var/www/html/release

[root@darksitewebcentos ~]# cd /var/www/html/release
4
UPDATING THE LCM FRAMEWORK
Updating the LCM Framework at a Dark Site
Update the LCM framework at a site without Internet access.

About this task

Update the LCM framework before attempting to update any other component.

Procedure

1. From a device that has public Internet access, go to the Nutanix portal and select
Downloads > LCM.

2. Next to the LCM Dark Site Bundle entry, click Download to download the latest LCM
framework tar file, lcm_dark_site_bundle_version.tgz.

3. Transfer the framework tar file to your local web server.

4. Extract the framework tar file into the release directory.

The following files are extracted into release.

• master_manifest.tgz

• master_manifest.tgz.sign

• modules

• nutanix_compatibility.tgz

• nutanix_compatibility.tgz.sign

• support.csv

• support.md

LCM |  Updating the LCM Framework | 13

5. Perform the following steps on each Nutanix cluster.

a. Log on to Prism.
b. Open LCM according to the procedure for the version of AOS you are using, as described
in Accessing the Life Cycle Manager from Prism Element on page 6 or Accessing the Life
Cycle Manager from Prism Central on page 7.
c. On the LCM page, click Settings.

d. Click the drop-down arrow in the Source field and select Local web server. In the URL
field, enter the path to the directory where you extracted the tar file on your local server.
End the URL with /release.

e. Click Save.
You return to the Life Cycle Manager.
f. Select Inventory in the LCM sidebar and click Perform Inventory.
The inventory operation automatically updates the framework.
5
LCM SOFTWARE UPDATES
Fetching the AHV Dark Site Bundle
Fetch and download the AHV dark site update bundle.

About this task

Procedure

1. From a device that has public Internet access, find the download links for versions of AHV,
and the compatibility and signature files, here.

2. Download the AHV tar file for the version you want to update to.

3. Download the compatibility and signature files.

• nutanix_compatibility.tgz

• nutanix_compatibility.tgz.sign

4. Transfer the tar file to your local web server.

5. On your local web server, extract the tar file into the release directory.
The following files are extracted into release/builds.

• /ahv-builds/version/metadata.json

• /ahv-builds/version/metadata.sign

• /ahv-builds/version/AHV-DVD-version.iso

6. In the release folder, replace the existing compatibility and signature files with the files that
you downloaded in step 3.

7. On each Nutanix cluster, perform an LCM inventory.

AHV now appears as an available update.

Figure 5: Software update: AHV

8. Continue to Updating Software at a Dark Site on page 17.

LCM |  LCM Software Updates | 15

Fetching the Foundation Dark Site Bundle
Fetch and download the Foundation dark site update bundle.

About this task

Procedure

1. From a device that has public Internet access, go to the Nutanix portal and select
Downloads > Foundation > Darksite.

2. Download the Foundation tar file, foundation_dark_site_bundle_version.tar.gz.

3. Download the compatibility and signature files.

• nutanix_compatibility.tgz

• nutanix_compatibility.tgz.sign

4. Transfer the files to your local web server.

5. On your local web server, extract the tar files into the release directory.
The following files are extracted into release/builds.

• /builds/foundation-builds/version/metadata.sign

• /builds/foundation-builds/version/foundation-version-NO_HX.tar.gz

• /builds/foundation-builds/version/metadata.json

6. In the release folder, replace the existing compatibility and signature files with the files that
you downloaded in step 3.

7. On each Nutanix cluster, perform an LCM inventory.

Foundation now appears as an available update.

Figure 6: Software update: Foundation

8. Continue to Updating Software at a Dark Site on page 17.

Fetching the Calm Dark Site Bundle

Fetch and download the Calm dark site update bundle.

About this task

Procedure

1. From a device that has public Internet access, go to the Nutanix portal and select
Downloads > Calm.

LCM |  LCM Software Updates | 16

 2. Choose your required version and download the tar files Nucalm-version.zip and
Epsilon-version.zip.

3. Download the compatibility and signature files.

• nutanix_compatibility.tgz

• nutanix_compatibility.tgz.sign

4. Transfer the files to your local web server.

5. On your local web server, extract Nucalm-version.zip and Epsilon-version.zip into the
release directory.
The following files are extracted into release/builds.

• /builds/calm-builds/version/metadata.json

• /builds/calm-builds/version/metadata.sign

• /builds/calm-builds/version/nucalm.tar.gz

• /builds/epsilon-builds/version/metadata.json

• /builds/epsilon-builds/version/metadata.sign

• /builds/epsilon-builds/version/epsilon.tar.gz

6. In the release folder, replace the existing compatibility and signature files with the files that
you downloaded in step 3.

7. On each Nutanix cluster, perform an LCM inventory.

Calm now appears as an available update.

8. Continue to Updating Software at a Dark Site on page 17.

Updating Software at a Dark Site

Use LCM to perform software updates at a dark site.

Before you begin

• Set up a local web server, as described in Setting Up a Local Web Server (Windows) on
page 9 or Setting Up a Local Web Server (Linux) on page 11.
• Update the LCM framework, as described in Updating the LCM Framework at a Dark Site on
page 13.
• Fetch and download the dark site update bundle for your software, as described in the
chapter for the software you want to update.

Procedure

1. Open LCM according to the procedure for the version of AOS you are using, as described
in Accessing the Life Cycle Manager from Prism Element on page 6 or Accessing the Life
Cycle Manager from Prism Central on page 7.

LCM |  LCM Software Updates | 17

2. In the LCM sidebar, under Updates, select Software.

3. Select the checkboxes for the updates you want to perform.

4. Click NCC Check.

5. In the dialog box that appears, specify which prechecks you want LCM to run before
updating and click Run.

6. When the prechecks are complete, click Update.

LCM |  LCM Software Updates | 18

7. Specify how you want LCM to handle pinned VMs.

a. ESXi: LCM cannot place ESXi hosts with active pinned VMs into maintenance mode.
Before you can proceed with LCM updates, you must choose one of the following.

• Select Automatically shut down non-migratable VMs through LCM.

• Uncheck Automatically shut down non-migratable VMs through LCM and handle
pinned VMs manually, either by migrating them or shutting them down.
• If you have a system configuration that can automatically handle pinned VMs,
uncheck Automatically shut down non-migratable VMs through LCM and leave your
system to handle them.

Figure 7: Applying updates on ESXi hosts

b. AHV: LCM automatically shuts down any running non-migratable VMs before updating
their host, and starts them again after completing the update.

Note: Non-migratable VMs include VMs with affinity set to a single host; VMs with GPUs;
and agent VMs.

8. Click Apply Updates.

LCM updates the selected components.

9. Repeat steps 2 through 8 on each Nutanix cluster.

10. To return to Prism, open the drop-down menu on the upper left and select Home.

LCM |  LCM Software Updates | 19

6
LCM FIRMWARE UPDATES
Dark Site Firmware Updates
Update firmware at a site without Internet access.

About this task

Procedure

1. From a device that has public Internet access, go to the Nutanix portal and select
Downloads > LCM.

2. Download the firmware tar file from the list of available bundles.

The firmware tar file uses the name lcm_darksite_firmware_manufacturer_version.tgz.

For example: lcm_darksite_firmware_nx_version.tgz

3. Transfer the tar file to your local web server.

4. On your local web server, extract the tar file into the release directory.

5. Open LCM according to the procedure for the version of AOS you are using, as described
in Accessing the Life Cycle Manager from Prism Element on page 6 or Accessing the Life
Cycle Manager from Prism Central on page 7.

LCM |  LCM Firmware Updates | 20

6. In the LCM sidebar, under Updates, select Firmware.

7. Select the updates you want to perform.

a. Select the checkbox for the node you want to update, or select All to update the entire
cluster.
b. Select the components you want to update. When you select a node, LCM selects
the checkboxes for all updateable components by default. Clear the checkbox of any
component you do not want to update.

8. Click NCC Check.

9. In the dialog box that appears, specify which prechecks you want LCM to run before
updating and click Run.

LCM |  LCM Firmware Updates | 21

10. When the prechecks are complete, click Update.

11. Specify how you want LCM to handle pinned VMs.

a. ESXi: LCM cannot place ESXi hosts with active pinned VMs into maintenance mode.
Before you can proceed with LCM updates, you must choose one of the following.

• Select Automatically shut down non-migratable VMs through LCM.

• Uncheck Automatically shut down non-migratable VMs through LCM and handle
pinned VMs manually, either by migrating them or shutting them down.
• If you have a system configuration that can automatically handle pinned VMs,
uncheck Automatically shut down non-migratable VMs through LCM and leave your
system to handle them.

Figure 8: Applying updates on ESXi hosts

b. AHV: LCM automatically shuts down any running non-migratable VMs before updating
their host, and starts them again after completing the update.

Note: Non-migratable VMs include VMs with affinity set to a single host; VMs with GPUs;
and agent VMs.

12. Click Apply Updates.

LCM updates the selected components.

13. Repeat steps 6 through 12 on each Nutanix cluster.

LCM |  LCM Firmware Updates | 22

 14. To return to Prism, open the drop-down menu on the upper left and select Home.

Effects of Updates on the Cluster

Effects on the cluster while LCM updates components.

LCM Workflow
LCM updates the cluster one node at a time: it brings a node down, performs updates, brings
the node up again, and then moves on to the next node. If LCM encounters a problem during an
update, it waits until you resolve the problem before moving on to the next node.
During an LCM update, there is never more than one node down at the same time.
For hosts running ESXi: LCM restarts the host after it migrates the guest VMs but before it
enters maintenance mode, so a "maintenance" status alert never appears in vCenter. This is
expected behavior.
All LCM updates follow the procedure shown in the following flowchart.

1. If updates for the LCM framework are available, LCM auto-updates its own framework, then
continues with the operation.
2. After updating itself, if necessary, LCM runs the series of pre-checks described in Life Cycle
Manager Pre-Checks on page 26.
3. When the pre-checks are complete, LCM looks at the available component updates and
batches them according to dependencies. LCM batches updates in order to reduce the
down-time of the cluster; when updates are batched, LCM only performs the pre-update
and post-update actions once. For example, on NX platforms, BIOS updates depend on BMC
updates, so LCM batches them so the BMC always updates before the BIOS on each node.
4. Next, LCM chooses a node and performs any necessary pre-update actions.

LCM |  LCM Firmware Updates | 23

5. Next, LCM performs the update. What actually happens during an update varies by
component; see the following sections for lists of update actions for supported components.
6. LCM performs any necessary post-update actions and brings the node back up.
7. When cluster data resiliency is back to Normal, LCM moves on to the next node.

Update Actions for Each Component

BIOS
What happens during the update:
1. Put the CVM in maintenance mode.
2. The CVM entering maintenance mode automatically triggers the host to migrate all
guest VMs to another node.
3. Restart the node into the Phoenix ISO.
4. Perform the first stage of the update.
5. Warm-reset Phoenix. (A warm reset does not power-cycle the node.)
6. Perform the second stage of the update.
7. Cold-reset Phoenix. (A cold reset power-cycles the node.)
8. Perform the third stage of the update.
9. Restart out of Phoenix and bring the CVM out of maintenance mode.

Note: Because of requirements of Intel microcode, updating the BIOS requires several
restarts, so BIOS updates take longer than updates for other components.

BMC
What happens during the update:
1. Put the CVM in maintenance mode.
2. The CVM entering maintenance mode automatically triggers the host to migrate all
guest VMs to another node.
3. Restart the node into the Phoenix ISO.
4. Perform the update.
5. Restart out of Phoenix and bring the CVM out of maintenance mode.
Data Drives and HBA Controllers
What happens during the update:
1. Check disk health on all CVMs, to make sure that taking down one drive does not
cause any data loss.
2. Move all storage traffic out of the Controller VM.
3. Put the CVM in maintenance mode.
4. The CVM entering maintenance mode automatically triggers the host to migrate all
guest VMs to another node.
5. Stop services on the CVM.
6. Restart the node into the Phoenix ISO.
7. Perform the firmware update.
8. Restart the node out of Phoenix.
9. Bring the CVM out of maintenance mode, automatically returning guest VMs.
10. Restart services on the CVM.
11. Return storage traffic to the CVM.
12. Recheck all data drives.
SATA DOM
What happens during the update:
1. Put the CVM in maintenance mode.

LCM |  LCM Firmware Updates | 24

 2. The CVM entering maintenance mode automatically triggers the host to migrate all
guest VMs to another node.
3. Restart the node into the Phoenix ISO.
4. Perform the update.
5. Restart out of Phoenix and bring the CVM out of maintenance mode.
M.2 Drive
What happens during the update:
1. Put the CVM in maintenance mode.
2. The CVM entering maintenance mode automatically triggers the host to migrate all
guest VMs to another node.
3. Restart the node into the Phoenix ISO.
4. Perform the update.
5. Restart out of Phoenix and bring the CVM out of maintenance mode.

LCM |  LCM Firmware Updates | 25

7
LCM PRECHECKS
Life Cycle Manager Pre-Checks
List of pre-checks that LCM performs during entity update.
LCM runs a series of pre-checks before beginning an inventory or update operation. If a check
fails, the operation does not run.
There are also prerequisites for certain components that fall outside the scope of these tests. If
your pre-check fails, make sure that you have met all these prerequisites:

• For all LCM operations, disable Common Access Card (CAC) authentication on your cluster.
• For all LCM updates on ESXi, disable admission control in vCenter.

Note: For information on configuring vCenter, see the vSphere Administration Guide for
Acropolis: vCenter Configuration.

LCM Pre-Checks
test_aos_afs_compatibility
Verifies that AOS and Nutanix Files are compatible.
test_cluster_status
Verifies that the cluster statuscommand returns a healthy state.
test_cassandra_status
Checks zeus_config_proto (run zeus_config_printer for equivalent) for the following:

• Node cassandra_status is not Normal.

• No nodes are in maintenance.
• The NCC check described in KB 4301 returns a PASS.
test_nutanix_partition_space
Checks that all CVMs have sufficient space on /home/nutanix.
test_network
Verifies that CVM IPs are reachable.
test_hypervisor_tests
Verifies that SSH can connect to the hypervisor.

• AHV: verifies libvirtd on AHV hosts.

• ESXi: checks if there is a VmkNic for all ESXi hosts in the same subnet as eth0/eth2.
Verifies pyVim connectivity to ESXi hypervisor.

LCM |  LCM Prechecks | 26

test_zk_status
Verifies that at least three zk hosts are available, on a cluster having three or more nodes.
test_replication_factor
Verifies that no containers in the cluster have RF=1.
test_under_replication
Checks for under-replication in the cluster.
test_prism_central_prism_element_compatibility
Checks that the registered Prism Central is compatible with Prism Element by reading
prism_central_compatibility.json.
test_prism_central_ova_compatibility
Checks that Prism Central has boot and data partitions on different disks.
test_degraded_node
Verifies that no nodes are degraded.
test_vlan_detection
Verifies that LCM can fetch VLAN information from all nodes using NodeManager.
test_upgrade_in_progress
Checks to see if a non-LCM upgrade is in progress.
test_foundation_service_status_in_cluster
Checks to see if the Foundation service is running on any node in the cluster.
test_vmotion_support/test_live_migration_support
Checks that an ESXi or Hyper-V host has vMotion enabled.
test_cluster_config
Verifies that the cluster has vSphere HA enabled on vCenter mapped to a Nutanix cluster.
(Make sure that you disable admission control.)
test_hosts_managed_by_same_mgmt_server
Verifies that the same management server (vCenter) manages all hosts.
test_hypervisor_config

• All hosts: verifies that the host has sufficient space.

• ESXi:

• Verifies that the host does not have APD/VMCP enabled. (If yes, disable it.)
• Verifies that no CD-ROMs (ISO image stored in local storage) or other physical
devices are attached to the VM. (If yes, remove the physical device from UVM.)
• AHV: verifies that the host can enter maintenance mode.
• Hyper-V: none.
test_check_revoke_shutdown_token
Checks that the CVM can revoke the shutdown token after an LCM update completes on
a node in a cluster.
test_ntp_server_reachability
Checks that the configured NTP servers are reachable before starting LCM updates.

LCM |  LCM Prechecks | 27

test_prism_central_minimum_resource_check
Check that Prism Central is configured with the correct license and has enough memory
to support all licensed features.
test_two_node_cluster_checks
Check that the configuration on a two-node cluster allows LCM updates.

• Check that both nodes are up and the cluster is functional.

• Check that the cluster has selected an LCM leader.
• Check that the cluster has configured a witness VM.
• Check that there are at least two meta disks available on each CVM.
test_zk_migration
Check to see if there is a Zookeeper migration in progress.
test_catalog_workflows
Check that the most commonly used catalog workflows for LCM are currently
performable.
test_foundation_workflows
Check whether the current version of Foundation on the cluster is on the LCM blacklist.
test_uniform_foundation_across_cluster
Check to make sure that every node in the cluster is running the same version of
Foundation.
test_esxi_bootbank_space
Check that the ESXi bootbank has enough space for LCM upgrades that involve
restarting into Phoenix.
test_esx_entering_mm_pinned_vms
Check to see if any node in the cluster contains a VM that cannot be live migrated to
another node.

Note: For inventory scans, LCM skips the aos_afs_compatibility and under_replication tests,
because inventory scans are not disruptive.
8
LCM LOG COLLECTION
Collecting Life Cycle Manager Logs
Run the LCM log collector utility.

About this task

The LCM log collector utility is a script independent of NCC, that collects only files that are
relevant to the LCM workflow. It completes more quickly and returns a smaller log bundle than
the NCC log_collector command.
The utility is independent of the state of LCM, so it can collect logs even when the LCM
framework is down.
The log collector utility performs the following actions:

• Collects LCM log files

• Detects if the node is stuck in Phoenix and collects kernel logs
• Collects logs for the services with which LCM interacts, such as Foundation and Prism
• Parses LCM configurations from zookeeper nodes.

Note: The collector cannot pull logs from a node booted into Phoenix when the node IP address
is not reachable over the network. In that case, apply the CVM IP address to the Phoenix instance
using IPMI, following the procedure in KB 5346, before running the collector.

Procedure
From any CVM, run the collector command.
$ python /home/nutanix/cluster/bin/lcm/lcm_log_collector.py
LCM creates a log bundle in the /home/nutanix directory on the node where you ran the script.
The log bundle format is lcm_logs_lcm-leader_timestamp.tar.gz.

LCM |  LCM Log Collection | 29

9
LCM GLOSSARY
Glossary
Glossary of LCM-related terms.

Hardware Terms
BMC
baseboard management controller, the microcontroller that manages the motherboard.
BIOS
basic input/output system, the firmware that initializes the motherboard and runtime
services on startup.
HBA
host bus adapter, a device that manages communication between storage media and other
system components.
SATA DOM
SATA disk on a module, the hypervisor boot drive for Nutanix platforms up to and including
G5.
M.2
mSATA 2, the hypervisor boot drive for Nutanix platforms G6 and later.

Dell-Specific Terms
iDRAC
integrated Dell remote access controller, a software tool that lets you administer a server
without needing physical access.
iSM
iDRAC service module, a module that integrates iDRAC with an operating system.
PTAgent
Power Tools agent, the software entity responsible for configuring the iSM.
Firmware Entities
A collective term for all Dell firmware not related to iDRAC.

Lenovo-Specific Terms
IMM2
integrated management module 2, a service processor that manages the motherboard
and allows remote access for server administration. Present on Lenovo generation-5
platforms; replaced by XCC in later generations.

LCM |  LCM Glossary | 30

UEFI
unified extensible firmware interface, the software entity that initializes the motherboard and
runtime services on startup. A replacement for the BIOS.
XCC
XClarity controller, a service processor that manages the motherboard and allows remote
access for server administration. Present on Lenovo platforms that use the Intel "Purley"
motherboard, replacing the IMM2 from earlier generations.

LCM |  LCM Glossary | 31

COPYRIGHT
Copyright 2020 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual
property laws. Nutanix and the Nutanix logo are registered trademarks of Nutanix, Inc. in the
United States and/or other jurisdictions. All other brand and product names mentioned herein
are for identification purposes only and may be trademarks of their respective holders.

License
The provision of this software to you does not grant any licenses or other rights under any
Microsoft patents with respect to anything other than the file server implementation portion of
the binaries for this software, including no licenses or any other rights in any hardware or any
devices or software that are used to communicate with or in connection with this software.

Conventions
Convention Description

variable_value The action depends on a value that is unique to your environment.

ncli> command The commands are executed in the Nutanix nCLI.

user@host$ command The commands are executed as a non-privileged user (such as

nutanix) in the system shell.

root@host# command The commands are executed as the root user in the vSphere or
Acropolis host shell.

> command The commands are executed in the Hyper-V host shell.

output The information is displayed as output from a command or in a

log file.

Default Cluster Credentials

Interface Target Username Password

Nutanix web console Nutanix Controller VM admin Nutanix/4u

vSphere Web Client ESXi host root nutanix/4u

vSphere Client ESXi host root nutanix/4u

SSH client or console ESXi host root nutanix/4u

SSH client or console AHV host root nutanix/4u

LCM | 
 Interface Target Username Password

SSH client or console Hyper-V host Administrator nutanix/4u

SSH client Nutanix Controller VM nutanix nutanix/4u

SSH client Nutanix Controller VM admin Nutanix/4u

IPMI web interface or Nutanix node ADMIN ADMIN

ipmitool

SSH client or console Acropolis OpenStack root admin

Services VM (Nutanix OVM)

SSH client or console Xtract VM nutanix nutanix/4u

SSH client or console Xplorer VM nutanix nutanix/4u

Version
Last modified: September 3, 2020 (2020-09-03T14:09:38-07:00)

LCM |