Linux Trouble-Tool

Advanced Tools for the Linux DBA
Kellyn Pot'Vin-Gorman, Technical Intelligence Manager, Delphix, Inc.

Moderated By: Johan Ludvig
Presenting Sponsors
Technical Assistance
If you require assistance Maximize your screen with Please fill in the short
during the session, type the zoom button on the evaluation following the
your inquiry into the top of the presentation session. It will appear in
question pane on the right window. your web browser.
side.
Thank You to Our Presenting Sponsors
Quest helps IT Professionals simplify administration

Empower users with new insights through familiar tools tasks so they can focus on the evolving needs of their
while balancing the need for IT to monitor and businesses. Combined with its commitment to help
manage user created content. Deliver access to all data companies protect today’s investment while planning
types across structured and unstructured sources. for the future, Quest continues to deliver the most
comprehensive solutions to monitor, manage, protect
and replicate database environments
IDERA designs powerful software with one SentryOne empowers Microsoft data professionals to
goal in mind – to solve customers’ most complex achieve breakthrough performance across physical,
challenges with easy-to-use solutions. IDERA’s award- virtual and cloud environments. We develop solutions
winning SQL Server database solutions and multi- to monitor, diagnose, and optimize SQL Server
platform database, application and cloud monitoring performance, including Plan Explorer, the query-tuning
tools ensure your business never slows down. tool with more than 100,000 downloads.
Supporting Sponsor
Attend PASS Summit to Grow Your Career
Connect with a global network of 250,000+ data professionals
PASS Summit is the largest conference for technical professionals who
leverage the Microsoft Data Platform.
November 6-9 | Seattle, WA
PASSsummit.com
The Community
Kellyn Pot'Vin-Gorman
Member of the Oak Table Network
Oracle ACE Director alumnus
Technical Intelligence Manager at Delphix
B dbakevlar.com Speaker
Speaker numerous technical conferences for Oracle, Big Data, DevOps
@DBAKevlar Testing and SQL Server
linkedin.com/in/kellynpotvin
Advanced Tools for the Linux DBA
Kellyn Pot'Vin-Gorman, Technical Intelligence Manager, Delphix, Inc.
Presenting Sponsors
LINUX PERFORMANCE IS
HUGE
Monitoring Tools
PIDSTAT
NETSTAT
DSTAT
NMON
LSOF
Common Arguments and Values
• Some of the utilities aren’t standard and may need to be installed.
• Yum and apt-get are your friend
• You may require sudo, (super user) privileges to install
• <command> -x(xx)
• Arguments are cap sensitive, (as is everything in Unix world.)
• <command> -x 5 5
• First value is interval in seconds
• Second value is how many results to return
• <command> -h
• Provides help information about the utility or command
9
PIDSTAT
• We all use IOSTAT, VMSTAT,
MEMSTAT, etc.
• Focusing on the PID and
collecting quick information can
eliminate the need to gather it
elsewhere.
• Statistics on a process, either
global or by PID
Global PID STATS and By PID
Inspecting IO for a User with PIDSTAT
Inspecting Page Faults and Memory
What is DSTAT?
• Monitoring tool for CPU, disk and

network activity
• Displays ongoing values in interval
until q(uit)
• To Install:
• $ yum install dstat –y
• Or
• $ sudo apt-get install dstat
17
dstat Utility
18
dstat Utility
• Dstat for one disk and only CPU info and disk reads:
$ dstat -cdl -D xvda1
$ dstat -d
19
What is NMON?
• information dump creation

related to cpu, memory, IO
or network.
• Install:
$ yum install nmon –y
• Use menu
• q<enter> to quit or
• <ctrl> c
20
nmon CPU
• Click C <enter>
21
nmon memory
• m<enter>
22
nmon NFS
• NFS<enter>
23
Knowing About Open Files
Knowing what files are open are important when waiting for a
database to close.
Awareness of access of files by a process that may not be
known
To identify what user has what files open
fuser/lsof Utilities
• Displays list of UNIX processes using file-system
# lsof /var
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
syslogd 350 root 5w VREG 222,5 0 440818 /var/adm/messag
syslogd 350 root 6w VREG 222,5 339098 6248 /var/log/syslog
cron 353 root cwd VDIR 222,5 512 254550 /var -- atjobs
LSOF
• LiSt of Files
• In our example, we want to see the files that are open in the
Temp directory.
• Although lsof +D with the directory name should display
them, Linux may not provide as specific data as I’d prefer.
• I want the child PID 17669, only Temp and I want it sorted
• Using grep, we can combat this:
$ lsof | grep Temp | grep 17669 | sort
Temp files, for child 17669, Sorted
Querying Open Datafiles
$ lsof -c sqlservr | grep mdf

sqlservr 7 root 84u REG 8,1 8388608 1050924
/var/opt/mssql/data/tempdb.mdf
/var/opt/mssql/data/master.mdf
/var/opt/mssql/data/model.mdf
/var/opt/mssql/data/msdbdata.mdf
What Files Are Opened by User and VM/Docker Usage
• $ lsof –u <username>
• Jsmith user is having issues working with an application.
• Query reports back, but the corresponding commit doesn’t occur.
• Believe that login is incorrectly configured.
Now a touch of a file in the directory shows this:
jsmith@e09627d558b7:/opt/mssql-tools/bin$ touch
test.txt
touch: cannot touch 'test.txt': Permission denied
What does LSOF show?
FUSER and Killing Users
• -k,--kill kill processes accessing the named file

• -w,--writeonly kill only processes with write access
$ fuser -i -k <port>/tcp
$ fuser –kw <filename>
Fuser –v –m <filename>
USER PID ACCESS COMMAND

/var/opt/mssql/scripts/push.sh:
root 29314 ....m bash
root 31503 ....m bash
jsmith 21963 …m bash
fuser –kw push.sh

What is SAR?
• System Activity Report
• Provides information about
• CPU
• Memory and virtual memory
• I/O
• Network
• Provided by the SYSSTAT package
• What if it’s missing?
• Install it as root:
$yum -y install sysstat
33
sar Utility
$ sar -u 5 5
Linux 2.6.32-573.22.1.el6.x86_64 (linuxsource.delphix.local) 11/28/2017 _x86_64_ (2 CPU)
06:15:31 PM CPU %user %nice %system %iowait %steal %idle

06:15:36 PM all 0.20 0.00 0.20 48.58 0.00 51.01
06:15:41 PM all 0.30 0.00 0.20 47.98 0.10 51.41
06:15:46 PM all 0.30 0.00 0.20 46.92 0.00 52.58
06:15:51 PM all 0.20 0.00 0.20 47.77 0.00 51.82
06:15:56 PM all 0.20 0.00 0.20 47.78 0.10 51.72
Average: all 0.24 0.00 0.20 47.81 0.04 51.71
sar Utility
$ sar -q 5 5
Linux 2.6.32-573.22.1.el6.x86_64 (linuxsource.delphix.local)
11/28/2017 _x86_64_ (2 CPU)
06:16:45 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

06:16:50 PM 0 190 1.37 1.28 1.30
06:16:55 PM 0 190 1.34 1.28 1.29
06:17:00 PM 0 190 1.31 1.27 1.29
06:17:05 PM 0 190 1.36 1.29 1.30
06:17:10 PM 0 190 1.34 1.28 1.30
Average: 0 190 1.34 1.28 1.30
What is SADC/SADF?
• sadc: back-end system activity data collector utility behind sar
• Collects specified system data at configured intervals
• Saves data to specified binary files
• sadf: system activity data formatter
• report on data collected by sadc/sar
• output to be consumed by other programs like awk, MS-Excel, etc
• Options for translating and displaying dates, numbers, text
• Output to plain text or XML
nmon CPU
• Click C <enter>
38
nmon memory
• m<enter>
39
nmon NFS
• NFS<enter>
40
Network Diagnostics
• ifconfig
• Configure/display network interfaces
• netstat/ss
• Display cumulative network statistics
• ping
• Send ICMP echo packets to network hosts
• traceroute
• Trace the route of ICMP echo packets to network hosts
• tcpdump/snoop
ifconfig Utility
• Configures network interfaces
• Configure new network interface: ifconfig … create inet … up
• Display network interface status: ifconfig -a
$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 06:8D:24:F2:82:3C
inet addr:10.0.1.20 Bcast:10.0.1.255
Mask:255.255.255.0
inet6 addr: fe80::48d:24ff:fef2:823c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:29480071 errors:0 dropped:0 overruns:0
frame:0
TX packets:26850299 errors:0 dropped:0 overruns:0
carrier:0
collisions:0 txqueuelen:1000
RX bytes:4059464946 (3.7 GiB) TX bytes:9135825067 (8.5
GiB)
Interrupt:153
What is NETSTAT?
• Displays network connections, routing tables, interface statistics
• Network connections (a.k.a. sockets): netstat
• Cumulative networking statistics: netstat -s
• Routing tables: netstat -r
• STREAMS statistics (Solaris and AIX): netstat -m
• Verbose mode: add -v switch
netstat Utility
$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 linuxsource:12129 linuxsource:ncube-lm ESTABLISHED
tcp 0 64 linuxsource:ssh c-164-182-219.hsd1:50171 ESTABLISHED
tcp 0 0 linuxsource:ncube-lm ip-10-0-1-10.delphix.:58424 ESTABLISHED
tcp 0 0 linuxsource:ncube-lm landsharkengine.delph:63611 ESTABLISHED
tcp 0 0 linuxsource:ncube-lm linuxsource:12129 ESTABLISHED
udp 0 0 localhost:17493 localhost:17493 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 12 [ ] DGRAM 8754 /dev/log
unix 2 [ ] DGRAM 7377 @/org/kernel/udev/udevd
unix 3 [ ] STREAM CONNECTED 9499
unix 2 [ ] DGRAM 9338
unix 2 [ ] DGRAM 8927
netstat Utility
• Display network routing tables on the local host:
$ netstat -r
Kernel IP routing table

Destination Gateway Genmask Flags MSS Window irtt
Iface
10.0.1.0 * 255.255.255.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth0
default 10.0.1.1 0.0.0.0 UG 0 0 0 eth0
What is SS?
• SS stands for Socket Statistics
• Works like NETSTAT
• Gathers info from the kernel via NAD
46
ss Utility
$ ss
$ ss -ltp
47
ping Utility
• Simple diagnostic test for network connectivity
• Be aware that many network administrator block ICMP traffic for security reasons
$ ping -c 4 www.amazon.com
PING e15316.ci.akamaiedge.net (104.86.87.164) 56(84) bytes of data.
64 bytes from a104-86-87-164.deploy.static.akamaitechnologies.com (104.86.87.164): icmp_seq=1
ttl=50 time=1.81 ms
ttl=50 time=1.83 ms
ttl=50 time=1.99 ms
ttl=50 time=1.85 ms
--- e15316.ci.akamaiedge.net ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.812/1.874/1.994/0.082 ms
What is TRACEROUTE?
• Display the route taken by ICMP packets to an IP host

• If missing, install using:
$yum install traceroute -y
49
traceroute Utility
$ traceroute www.amazon.com
traceroute to www.amazon.com (54.192.195.66), 30 hops max, 60 byte packets
1 216.182.225.134 (216.182.225.134) 16.379 ms 216.182.225.124 (216.182.225.124) 49.305 ms
216.182.225.134 (216.182.225.134) 16.347 ms
2 100.66.8.70 (100.66.8.70) 19.966 ms 100.66.8.6 (100.66.8.6) 19.962 ms 100.66.8.250
(100.66.8.250) 14.142 ms
3 100.66.10.46 (100.66.10.46) 21.317 ms 100.66.10.100 (100.66.10.100) 19.104 ms 100.66.11.150
(100.66.11.150) 14.317 ms
4 100.66.7.237 (100.66.7.237) 15.802 ms 100.66.7.183 (100.66.7.183) 16.916 ms 100.66.6.67
(100.66.6.67) 16.800 ms
5 100.66.4.199 (100.66.4.199) 19.609 ms 100.66.4.127 (100.66.4.127) 14.576 ms 100.66.4.151
(100.66.4.151) 11.455 ms
6 100.65.9.65 (100.65.9.65) 0.349 ms 100.65.9.33 (100.65.9.33) 5.948 ms 100.65.9.1 (100.65.9.1)
0.275 ms
7 205.251.244.243 (205.251.244.243) 1.378 ms 72.21.220.120 (72.21.220.120) 1.280 ms 1.333 ms
8 54.239.111.120 (54.239.111.120) 24.074 ms 23.301 ms 54.239.110.22 (54.239.110.22) 37.663 ms
9 54.239.111.183 (54.239.111.183) 1.415 ms 54.239.111.179 (54.239.111.179) 1.234 ms 1.280 ms
10 72.21.220.9 (72.21.220.9) 1.326 ms 1.348 ms 1.391 ms
iPerf3 Utility
• iPerf3 is a tool for active measurements of the maximum achievable
bandwidth on IP networks
• supports tuning of various parameters related to timing, buffers and protocols
(TCP, UDP, SCTP with IPv4 and IPv6)
• Home page https://iperf.fr/
• “iperf” executable must be installed on servers on both ends of
network being measured
Server-A Server-B
(iperf client) (iperf server)
iperf –p 50001 –c Server-B –l 8192 –t 60 –P 32 iperf –p 50001 –s

What is TCPDUMP?
• Dump/sniff traffic on a network
• Another one that may be missing:
$ yum install tcpdump -y
52
tcpdump/snoop Utilities
$ tcpdump
tcpdump: listening on hme0
18:56:08.427002 IP linuxsource.ssh > c-67-164-182-219.hsd1.co.comcast.net.50171:
Flags [P.], seq 529952:530160, ack 3681, win 172, length 208
18:56:08.428001 IP linuxsource.ssh > c-67-164-182-219.hsd1.co.comcast.net.50171:
18:56:08.429695 IP c-67-164-182-219.hsd1.co.comcast.net.50171 > linuxsource.ssh:
Flags [.], ack 515360, win 256, length 0
Flags [.], ack 517984, win 252, length 0
^C18:56:08.429925 IP linuxsource.ssh > c-67-164-182-219.hsd1.co.comcast.net.50171:
2781 packets captured

2781 packets received by filter
0 packets dropped by kernel
Trace Utilities
• Different utilities on different platforms, but for Linux:
• strace
• Attach to or run a process and then trace:
• UNIX system calls executed
• Signals received
• Machine faults incurred
• (optional) entry/exit trace of user level function calls
What is STRACE?
• Output from strace on Red Hat Enterprise Linux (RHEL)
• Oracle server process using SQL Trace
55
strace Utility
15:26:02.348478 gettimeofday({1003530362, 348525}, NULL) = 0

15:26:02.348659 pread(409,
"\6\2\0\0\240\27\200\0\325\227\7\0\0\0\2\0\0\0\0\0\1\0\7\0_\f\0\0\210I\7\0\0\0\24P\
2\6\3\0\30/\200\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\236\0\0\0N\1\354\3R\6R\6\0\0\236\0\1\0\2\0
\3\0\4\0\5\0"...,
8192, 12386304) = 8192
15:26:02.349260 write(6, "WAIT #1: nam=\'db file sequential read\' ela= 0 p1=2
p2=6048 p3=1", 63)
What is DTRACE?
• Performance analysis and troubleshooting tool (http://dtrace.org )
• Not just user-level software (i.e. applications, databases and webservers)
• also OS kernel and device drivers
• provides a language, (i.e. “D”) for writing DTrace scripts and one-liners
• Terrific case-study of I/O performance analysis using DTrace

• https://blogs.oracle.com/BestPerf/entry/i_o_analysis_using_dtrace
57
dtrace Utility
$ dtrace -n 'proc:::exec-success{printf("%d %s”,timestamp,curpsinfo-
>pr_psargs);}’
dtrace: description 'proc:::exec-success ' matched 1 probe
CPU ID FUNCTION:NAME
1 797 exec_common:exec-success 21935388676181394 man ls
0 797 exec_common:exec-success 21935388840101743 sh -c cd /usr/share/man;
tbl /usr/sha
1 797 exec_common:exec-success 21935388858652639 col –x
0 797 exec_common:exec-success 21935388863714971 neqn
/usr/share/lib/pub/eqnchar –
0 797 exec_common:exec-success 21935388867119787 tbl
/usr/share/man/man1/ls.1
1 797 exec_common:exec-success 21935388881310626 nroff -u0 -Tlp -man –
file Utility
“Finding Java”
$ file /usr/bin/java
/usr/bin/java: symbolic link to `/etc/alternatives/java’
$ file /etc/alternatives/java
/etc/alternatives/java: symbolic link to `/usr/java/jdk1.7.0_51/jre/bin/java’
$ file /usr/java/jdk1.7.0_51/jre/bin/java
/usr/java/jdk1.7.0_51/jre/bin/java: ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not
stripped
$ file make_logs.sql
make_logs.sql: ASCII C program text
FTrace
Perf-Events
eBPF
LTTng
SystemTap
KTap
Administrator
Perf Tools sysdig
About Linux Trace Files
• Calls are returned
• -1 is commonly a sign of an error
• Calls can be unfinished and then resumed
• Calls are kept as close to readable as possible, Example:
''ls-l /dev/null'' is captured as:

lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
TRACE-CMD
Perf-Tools
DIG
DeeP
Performance
and Tracing DXToolkit
• Not to duplicate previous in-
depth sessions
• Valuable and make you want to
learn more
• Easy to use
• Variety of tools across resources
• Hit main pain points
Goal
Why TRACE-CMD and Perf-Tools
• Uses FTRACE and Linux Perf Events

• Removes a huge percentage of tracing
and performance tools with numerous
overhead/knowledge.
• Already enabled and in use for most
Linux hosts
• Part of Linux kernel
Ftrace and the Case for Trace-CMD
• Great Github page, already installed on most systems, if not,
consider leaving this to a full admin vs. a DBA.
• Requires root privileges, setup of debugging and tracing on the
host, along with mounted of /sys volume.
• If you want to run this in a container, it is possible, but the container
must be created with privileged:true upon run.
• Uses the /sys/kernel/debug/tracing
• Consider simplifying using trace-cmd package for DBA use.
CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE
The Next two tools Use
Trace points, i.e kernel static tracing
Kernel dynamic tracing, aka kprobes
And User level tracing, also called uprobes
Produce kernel level trace output

In or Out
In the kernel or outside the kernel, it’s pretty simple.
• In-tree: ftrace, perf_events, eBPF

• Out-of-tree: System Tap, ktap, LTTng, dtrace4linux, Dtrace, sysdig
What are Tracepoints?
Probes
Trace Cmd
# trace-cmd record -e ext4

ls [...] # trace-cmd report version = 6 CPU 1 is empty cpus=2
trace-cmd-7374 [000] 1062.484227: ext4_request_inode: \ dev 253:2 dir 40801 mode 33188
trace-cmd-7374 [000] 1062.484309: ext4_allocate_inode: \ dev 253:2 ino 10454 dir 40801 mode 33188
Output file is trace.dat
Output is saved as a trace.dat file.

Example of Output
trace-cmd report trace-cmd-16129 [002] 158126.498411: function: __mutex_unlock_slowpath <--
mutex_unlock
trace-cmd-16131 [000] 158126.498411: kmem_cache_alloc: call_site=811223c5
ptr=0xffff88003ecf2b40 bytes_req=272 bytes_alloc=320 gfp_flags=GFP_KERNEL|GFP_ZERO
trace-cmd-16130 [003] 158126.498411: function: do_splice_to <-- sys_splice sleep-16133 [001]
158126.498412: function: inotify_inode_queue_event <-- vfs_write
trace-cmd-16129 [002] 158126.498420: lock_release: 0xffff88003f1fa4f8 &sb->s_type->i_mutex_key
trace-cmd-16131 [000] 158126.498421: function: security_file_alloc <-- get_empty_filp sleep-16133
[001] 158126.498422: function: __fsnotify_parent <-- vfs_write
trace-cmd-16130 [003] 158126.498422: function: rw_verify_area <-- do_splice_to
trace-cmd-16131 [000] 158126.498424: function: cap_file_alloc_security <-- security_file_alloc
trace-cmd-16129 [002] 158126.498425: function: syscall_trace_leave <-- int_check_syscall_exit_work
sleep-16133 [001] 158126.498426: function: inotify_dentry_parent_queue_event <-- vfs_write [003]
158126.498426: function: security_file_permission <-- rw_verify_area
trace-cmd-16129 [002] 158126.498428: function: audit_syscall_exit <-- syscall_trace_leave
Reading a Filtered Ftrace Report
• CPU=0
• Success/Error/Warning/Info
• Success = 1
# trace-cmd record -N gandalf:12345 -e sched_switch -e sched_wakeup -e irq hackbench 50
Network Tracing with Tracer-Cmd

• Network trace from
FRODO host to
GANDOLF host
• File is created on target,
GANDOLF
• Can view exactly what
transpires during the
network connection
Linux Observatory Perf Tools
• Function collectors
• Funccount
• Functrace
• Funcslower
• Funcgraph
• kprobe
Snoop
• IO Opensnoop
• Iosnoop Execsnoop
• Iolatency syscount
• Bitesize
https://github.com/brendangregg/perf-tools
Perf-Tools
• Sources the Linux kernel

• Easily added via the linux-tools-common
• Dynamic buffering means very little overhead to collect
performance data
• Capable of profiling, CPU performance, user-level stack collection
• Capable of consuming debug info for line tracing and local
variables
• No kernel programming at this time and very safe to install
IOSnoop
• By Device
• By ioType
• By Name
• Includes
• Start/end time
• Usage
• Duration
• Block info
• Queueing time
• CPU Time
IOSNOOP- Workload Differences
Standard Disk
COMM PID TYPE DEV BLOCK BYTES LATms
randread 6199 R 202,16 71136208 8192 5.32
randread 6199 R 202,16 83134400 8192 9.26
randread 6199 R 202,16 88477928 8192 3.46
randread 6199 R 202,16 66953696 8192 10.69
randread 6199 R 202,16 87832704 8192 3.68
randread 6199 R 202,16 74963120 8192 4.62
SSD
COMM PID TYPE DEV BLOCK BYTES LATms
randread 20125 R 202,16 24803976 8192 0.15
randread 20125 R 202,32 17527272 8192 0.15
randread 20125 R 202,16 13382360 8192 0.15
randread 20125 R 202,32 29727160 8192 0.19
randread 20125 R 202,32 26965272 8192 0.18
randread 20125 R 202,32 27222376 8192 0.17
Outliers in IO Data
• Function vs. Utility
• Executed printk()
• C programming language against
the Linux Kernel
• Prints the string to kernel log,
allowing you to post messages
directly.
PrintK
Recent Changes
Recent changes in the /etc/sysctl.conf. Now these numbers

may not seem very important between the versions:
kernel.printk = 4 4 1 7  kernel.printk = 3 4 1 3
These actually correspond to:

• console_loglevel
• default_message_loglevel
• minimum_console_loglevel
• default_console_loglevel
PrintK Message - Older
KERN_EMERG /* system is unusable */
KERN_ALERT /* action must be taken immediately */
KERN_CRIT /* critical conditions */
KERN_ERR /* error conditions */
KERN_WARNING /* warning conditions */
KERN_NOTICE /* normal but significant condition */
KERN_INFO /* informational */
KERN_DEBUG /* debug-level messages */
• Use instead of atop
• Inspects short lived CPU processes
for demands on resources at a
kernel level
• Uses a “live log” approach for
historical research
Ever wanted to know exactly what
was happening during a start up of
an executable or while an
application is performing a task?
Execsnoop
Execution Tracing
Capture Unexpected Behavior
TIMESTAMPS DISPLAY EACH STEP IN AN APPLICATION PROCESS
• Similar to LSOF, focusing on open
files
• Doesn’t use strace, which is
sourced for LSOF
• More efficient
OPENsnoop
Reveals Same data as LSOF
Benefit over LSOF
Displays trace info on “files not found”

OK, the My Little Pony and Linux Tracing?
DIG
$ dig <host name> -<x> +<command>
Can be used with .digrc file on host to provide more information about
DNS.
Can use it with monitoring and alerting for the DBA to know when hosts
are experiencing DNS issues, which often are blamed on databases vs. DNS.
DIG Examples
Linux for the DBA Summary
• Utilities are powerful and provide incredible value to monitoring

and trouble shooting.
• Different solutions will be installed depending on the provider or
administrator’s preferences. Become familiar with the power of
each.
• There are utilities out there that have display options that are
similar to a GUI and output that can be used for reporting.
96
Coming up next…
Power BI for Master Data Management

Swagatika Sarangi
THANK YOU
FOR ATTENDING
@sqlpass
#sqlpass
@PASScommunity
Presenting Sponsors

Linux Trouble-Tool

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Linux Trouble-Tool

Hochgeladen von

Copyright:

Advanced Tools for the Linux DBA

Kellyn Pot'Vin-Gorman, Technical Intelligence Manager, Delphix, Inc.

Quest helps IT Professionals simplify administration

November 6-9 | Seattle, WA

Member of the Oak Table Network

Oracle ACE Director alumnus

Technical Intelligence Manager at Delphix

• Monitoring tool for CPU, disk and

• information dump creation

$ lsof -c sqlservr | grep mdf

• -k,--kill kill processes accessing the named file

USER PID ACCESS COMMAND

fuser –kw push.sh

$yum -y install sysstat

06:15:31 PM CPU %user %nice %system %iowait %steal %idle

06:16:45 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

Kernel IP routing table

--- e15316.ci.akamaiedge.net ping statistics ---

• Display the route taken by ICMP packets to an IP host

iperf –p 50001 –c Server-B –l 8192 –t 60 –P 32 iperf –p 50001 –s

2781 packets captured

15:26:02.348478 gettimeofday({1003530362, 348525}, NULL) = 0

• Terrific case-study of I/O performance analysis using DTrace

''ls-l /dev/null'' is captured as:

• Uses FTRACE and Linux Perf Events

Trace points, i.e kernel static tracing

Kernel dynamic tracing, aka kprobes

And User level tracing, also called uprobes

Produce kernel level trace output

In the kernel or outside the kernel, it’s pretty simple.

• In-tree: ftrace, perf_events, eBPF

# trace-cmd record -e ext4

Output is saved as a trace.dat file.

Network Tracing with Tracer-Cmd

• Sources the Linux kernel

Recent changes in the /etc/sysctl.conf. Now these numbers

These actually correspond to:

Displays trace info on “files not found”

• Utilities are powerful and provide incredible value to monitoring

Power BI for Master Data Management

Das könnte Ihnen auch gefallen