SER2965BU

Advanced Troubleshooting of i b u t i o n
is t r
ESXi Server 6.x for vSphere o r d
t i o n
Gurus b l i c a
r p u
t f o
:N o
t e n t
C on
1 7
r ld 20
Mw o
V

Ragavendra P Kumar & Abhilash Kunhappan

#VMworld #SER2965BU
Disclaimer
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product. t i o n
tribu
r dis purchase orders, or
• Features are subject to change, and must not be included in contracts,
o
sales agreements of any kind.
a t i on
c u b li
• Technical feasibility and market demand will affect o p
r final delivery.
o t f
• n
Pricing and packaging for any new technologiest : N or features discussed or presented have not
ont e
been determined.
17 C
2 0
o r ld
VMw

#SER2965BU CONFIDENTIAL 2
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 3
 t i o n
ibu

chmod 777 <user>

i s tr
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 4
Do You Know? 1. Configuration Files?

t i o n
2. Commands? i s tr ibu
or d
t ion 3. Log Files?
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

in ESXi Server

#SER2965BU CONFIDENTIAL 5
Learner Objectives
• Improvise troubleshooting steps for a vSphere admin by

• Understanding the ESXi Architecture from a component breakdown perspective

t i o n
i s tr ibu
• Learn “7” important and key :- or d
a t i on
Log Files : which log files to review and when u b lic
1.
o r p
o t f
2. ESXi Commands t : N issues
: to isolate and troubleshoot
n
o n te
Configuration Files : forms17 theCbackbone of troubleshooting sessions
3.
d 2 0
w o rl
VM
Every VMworld session is time bound and we will cover as much as we can.
For all remaining questions, we are always a phone call away.

#SER2965BU CONFIDENTIAL 6
Agenda
© 

t i o n
1 7 Important Log files
i s tr ibu
or d
2 t ion
7 Key ESXi Commands
bli c a
r p u
3 7 Significant Configuration Files Not fo
e n t :
C ont
1 7
r ld 20
Mw o
V

#SER2965BU CONFIDENTIAL 7
ESXi – Architecture
HTML 5 - vSphere
WebClient vpxd DRS
VIM Clients VIM FT
HA Guest OS & Apps
HTML 5 – ESXi Power
CLI
t i o n
ibu
Host Client vCPU
vCenter Server
i s tr
or d
vpxa
t ion
Hostd Sys
bli c a
FDM
Calls
p
User world Support
r u
VMX
o t fo
esxcli Process
nt: N
localcli
n e
tResource

VMK Call
Shell User worlds
C o Scheduler
VMM

d 2017 Monitor
l
vNIC

or Network Storage

VMw
Stack Stack vSCSI
vSCSI
DVFilter

vmkapi and Interrupt handler

VMkernel

Server Hardware

8
#SER2965BU CONFIDENTIAL
ESXi Log Files


t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER2965BU CONFIDENTIAL 9
1. vmksummary.log – Abrupt Host reboots
• Time stamp indicating ESXi server reboot

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
fo
Not shows host rebooting has not been tracked
• VMkernel will always track Host reboots, the below
n t :
ont e
1 7 C
r ld 20
Mw o
V

• Disable the Automated Server Recovery feature from the BIOS to avoid automatic reboots

#SER2965BU CONFIDENTIAL 10
2. boot.gz log - Slow Boot Issues
• To troubleshoot both slow boot and boot process hang of ESXi server

• Location : /var/log/boot.gz n
t i o
ibu i s t r
• Use DCUI - Alt + F12 to review VMkernel messages during this time o r d
t io n
li c a
bscreen
•
r u
Enabling Serial logging using (Shift + o) at the start pup for log redirection
o t fo
– vmkBootVerbose=TRUE vmbLog=TRUEnt : N
logPort=com1
o n te
1 7 C no-auto-partition bootUUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
d 2 0
– debugLogToSerial=1 logPort=com1

w o rl
VM

#SER2965BU CONFIDENTIAL 11
2. boot.gz log - Slow Boot Issues
• Slow ESXi Boot Issues when, RDM devices are not perennially tagged
• Validate the esx.conf file with a key word search “perennialResvd” with respect to the NAA.ID of
the LUN t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
o rld
w perennially reserve tag per LUN
VMthe
• Command to set
# esxcli storage core device setconfig -d naa.id --perennially-reserved=true

#SER2965BU CONFIDENTIAL 12
3. hostd & hostd-probe.log – ESXi Not Responding
• hostd unresponsive and Host Not responding from vCenter server
• A successful hostd process start up or working fine hostd process should finish reading and
populating the “vmInventory.xml” file t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M
• Direct Connect to the ESXi with Host client should also validate the hostd status
• Do NOT restart the ESXi Server when hostd is in a loop if you need to pursue the cause

#SER2965BU CONFIDENTIAL 13
3. hostd.log – ESXi Not Responding
• Understanding the Hostd status and validating the Hostd service restart is successful

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
t fo
No
• Key words search in hostd.log – “BEGINt:SERVICES”
n
n t e
17 Co
2 0
w o rld
V M

#SER2965BU CONFIDENTIAL 14
4. vmware.log - Virtual Machine Issues
• VM Power ON issues
• Attempt a Virtual Machine power ON and then validate the below log
t i o n
• Identify the cause of the Power ON failure
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER2965BU CONFIDENTIAL 15
5. vmkernel.log - Storage Issues
Understanding SCSI Sense codes are key for Storage issues

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
1 7 C
2 0
w o rld
V M
• SCSI codes which are T10 org defined codes
http://www.t10.org/lists/2op.htm
• SCSI sense codes
http://www.virten.net/vmware/esxi-scsi-sense-code-decoder/

#SER2965BU CONFIDENTIAL 16
6. vobd.log - Network & Storage Issues
• Network showing uplink transition

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
• Storage disconnect messages
o n te
1 7 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 17
7. fdm.log – High Availability (HA) Issues

to identify :-
• the HostID?
t i o n
• the Host is a Slave or Master?
i s tr ibu
• the Host that was isolated or d
t ion
from cluster?
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 18
ESXi Commands
©

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 19
1. To Monitor & Configure ESXi Server Settings…
boot vib
module 03 04 profile
process sources t i o n
i s tr ibu
or d
nic t ion
bli c a
firewall 02
r p u cluster
vSwitch t f o 05 network
: N o
e n t datastore
C ont
1 7
r ld 20
Mw o
san V list
vvol 01 06 kill
vmfs process
esxcli

#SER2965BU CONFIDENTIAL 20
# esxcli Displaying the ARP
and Neighbor
Discovery cache

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
VMX World
o t fo
ID
nt: N
information VMX Cartel
o n te
1 7 C
ID
r ld 20information
Mw o
V

#SER2965BU CONFIDENTIAL 21
2. VMkernel Sysinfo Shell Command…
** # vsish
get <opt.1>/<opt.2>/<opt.3>,….
t i o n
i s tr ibu
or d
t ion
bli c a
hardware netfor pu storage
n t : Not
ont e
1 7 C
r ld 20
Mw o
V
/bios /pNics/vmnic<>/stats /scsifw/devices/.<>/stats
/hardwareInfo /portsets/vSwitch<>/stats /scsifw/adapters/.<>/stats

** take assistance always from GSS when in doubt #SER2965BU CONFIDENTIAL 22

# vsish

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo Get the VMX
nt: N
onte
Cartel ID

1 7 C
r ld 20
Mw o
V Get the
GroupID

#SER2965BU CONFIDENTIAL 23
 3. To Manage ESXi & VM Configurations…
login auth
connect 03 04 task_list
registervm task_cancel
t i o n
i s tr ibu
or d
net t ion
bli c a port_info
r pu
storage 02
firmware
o t fo 05 service_list

n t : N remove_service
ont e
1 7 C
r ld 20
Mw o
getallvms V vprobes
power.getstate 01 06 shutdown
get.guest refresh_pnic
vim-cmd

#SER2965BU CONFIDENTIAL 24
# vim-cmd
Validates if the Hostd
is running by listing
the VM’s registered

t i o n
i s tr ibu
or d
a t i on
p ublic List the tasks running
fo r on the ESXi host
n t : Not
ont e
1 7 C
r ld 20
Mw o
V

#SER2965BU CONFIDENTIAL 25
4. To Manage VMFS Volumes & Virtual Disks…

# vmkfstools <opt.1> name.vmdk <opt.2>

t i o n
i s t r ibu
or d
t ion
bli c a -x
r pu
-e -t10 .

-v10 o t fo check,
n t : N-v10 repair
ont e
1 7 C
r ld 20
Mw o
V
to check VM to check Inode to check & correct
Snapshot Chain integrity of a vmdk basic file level
consistency like file in VMFS layer issues of a vmdk
PID & CID in and not in GOS file in VMFS layer
vmdks level

#SER2965BU CONFIDENTIAL 26
# vmkfstools VM Snapshot Chain
Consistency check

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
7 C Check or Repair
r ld 201 the Virtual Disk
Mw o
V

#SER2965BU CONFIDENTIAL 27
5. # memstats – to Get Detailed Memory Statistics
• VM is identified by the VMX cartel ID using the command ps | grep –i <vmx name>
• Memstats –r vm-stats –s name:<choose the options needed>

Get the VMX cartel

ID t i o n
i s tr ibu
Virtual Machine Stats check o r d
t n
io ID
based on the VMX
bli c acartel
p u
t f or
: N o
n tent
C o
0 17
rld 2
Mw o
V

#SER2965BU CONFIDENTIAL 28
# memstats
• VM is identified by the VMX cartel ID using the command ps | grep –i <vmx name>
• Memstats –r swap-stats –s name:<choose the options needed> -u mb
Swap Statistics on the
Virtual Machine
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER2965BU CONFIDENTIAL 29
6. # pktcap-uw - Network Packet Capture Analysis Tool
• pktcap-uw --trace -- vmk <vmk port> Output can be redirected to
• pktcap-uw --trace -- uplink <uplink name> a file as well

t i o n
i sibu
tr path from
o d
Shows packet traverse
r to identify drop
io
each layer
t n and
c a
bli within the ESXi host
r p u
o t fo
nt: N
o n te MAC address of the Port
17 C that is being captured
2 0
w o rld
V M

#SER2965BU CONFIDENTIAL 30
7. To Monitor Core ESXi Performance…
virtual machine
v
Power
(usage & management) CPU t i o n
p c
i s tr ibu
or d
t ion
bli c a
r p u
o tfo
network n N
nt:esxtop m
onte
memory
(vSwitch, uplinks)
1 7 C
r ld 20
Mw o
V
x d
vSAN
disk
u (hba level info)
storage
(LUN specific) #SER2965BU CONFIDENTIAL 31
# esxtop

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 32
ESXi – 7 Effective Commands for Troubleshooting
1. esxcli : Monitoring or configuring ESXi infra settings

2. vim-cmd
on
: Managing ESXi server & Virtual Machine related configurations
u ti
i b
r distr
3. vsish : VMkernel Sys Info Shell
on o
c a t i
ubli
or p& virtual disks from command line
4. vmkfstools : for management of VMFS volumes
t f
:N o
5. memstats : Memory statisticsonten
t
1 7 C
20
6. pktcap-uw orld
: Anwenhanced Network packet capture analysis tool
VM
7. esxtop : Real time performance monitoring

#SER2965BU CONFIDENTIAL 33
ESXi Configuration Files


t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 34
7 Configuration File
1. /etc/vmware/esx.conf : Holds information about Storage, Network and Hardware

2. /etc/vmware/hostd/vmInventory.xml : Virtual Machine Inventory list update on the ESXi

t i o n
ribu
3. /etc/vmware/hostd/authorization.xml dis–tauthorization
: vCenter to ESXi host connection
o r
o n
4. /etc/vmware/vpxa/vpxa.cfg : vCenter and ESXi cati
ubli connectivity
o r p
N tf
o configuration file
5. /etc/vmware/vmkiscsid/iscsid.conf
nt
: :
iSCSI
o n te
17 C
6. /etc/vmware/fdm 2 0 : FDM config file with Clusterconfig, Hostlist and VM metadata
w orld
7. V M
/etc/vmware/license.cfg : License configuration file for ESXi host

#SER2965BU CONFIDENTIAL 35
Q&A

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER2965BU CONFIDENTIAL 36
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M