Shikhar Verma 1

Kdump is a kernel feature which is used to capture crash dumps when the
system or kernel crash. For enabling kdump we have to reserve some portion
of physical RAM which will be used to execute kdump kernel in the event of
kernel panic or crash.

When a kernel crash or kernel panic occurs then running kernel runs
‘kexec(kdump kernel)‘ and it loads kdump kernel from reserve memory and
then contents of RAM and Swap is copied to vmcore file either on local disk or
on remote disk and finally reboot the box.

By analyzing the crash dumps we can find the reason or the root case of
system failure. If you have OS support then you can share the crash dumps to
the vendor for analysis.

2
Step:1 Install ‘kexec-tools’ using yum command

[root@cloud ~]# yum install kexec-tools

Step:2 Update the GRUB2 file to Reserve Memory for Kdump kernel

Edit the GRUB2 file (/etc/default/grub), add the parameter

‘crashkernel=<Reserved_size_of_RAM>‘ in the line beginning with
‘GRUB_CMDLINE_LINUX‘

Execute the below command to regenerate grub2 configuration.

[root@cloud ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

3
 In case of UEFI firmware, use the below command

 # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

 Above command will inform bootlaoder to reserve 128 MB RAM after reboot.

 Reboot the box now using below command :

 # shutdown -r now

4
 Step:3 Update the dump location & default action in the file
(/etc/kdump.conf)

 To store crash dump or vmcore file on a local file system, edit the
file ‘/etc/kdump.conf‘ and specify the location as per your setup.
In my case i am using a separate local file system ( /var/crash).
It is recommended that size of file system should be equivalent to
the size of your system’s RAM or file system should have free
space equivalent to the size of RAM. Kdump allows to compress
the dump data using ‘core collector’ option (core_collector
makedumpfile -c ) where -c is used for compression.

5
 In case if kdump fails to store the dump file to specified location
then default action will be performed which is mention in the default
directive. In my case default action is reboot.

 Update the below three directives in kdump.conf file.

 [root@cloud ~]# vi /etc/kdump.conf

 path /var/crash
 core_collector makedumpfile -c
 default reboot

6
 Step:4 Start and enable kdump service

# systemctl start kdump.service

# systemctl enable kdump.service

 Step:5 Now Test Kdump by manually crashing the system

Before crashing your system , please verify whether the kdump service is
running or not using below command.

# systemctl is-active kdump.service

# service kdump status

7
 To test our kdump configuration we will manually crash our
system with below commands.

# echo 1 > /proc/sys/kernel/sysrq ; echo c > /proc/sysrq-trigger

 This will create a crash dump file (vmcore ) under ‘/var/crash‘ file
system.
# ls -lR /var/crash

-rw-------. 1 root root 139147524 Mar 4 03:02 vmcore

-rw-r--r--. 1 root root 35640 Mar 4 03:02 vmcore-dmesg.txt

8
 Step:6 Use ‘crash’ command to analyze and debug crash dumps

Crash is the utility or command to debug and analyze the crash dump or vmcore file.

To use the crash, make sure two packages are installed : ‘crash & kernel-debuginfo‘

# yum install crash

To install ‘kernel-debuginfo’ package , first enable debug repo. Edit the repo file
/etc/yum.repos.d/CentOS-Debuginfo.repo

change ‘enbled=0’ to ‘enabled=1’

9
 yum install kernel-debuginfo

 Once the kernel-debuginfo is installed , then try to execute below

crash command, it will give us a crash prompt where we can run
commands to find process info , list of open files when the system
got crashed.

 # crash /var/crash/127.0.0.1-2016-03-04-14\:20\:06/vmcore
/usr/lib/debug/lib/modules/`uname -r`/vmlinux

 crash>

10
 Type ‘ps‘ command to list the Process which were running when the system
got crashed.

 crash> ps

 To view the files that were open when system got crashed , type ‘files’
command at crash prompt.

 crash> files
 PID: 5577 TASK: ffff88007b44f300 CPU: 0 COMMAND: "bash"
 ROOT: / CWD: /root
 FD FILE DENTRY INODE TYPE PATH

11
END of this Course Module.

Thanks
12