Sie sind auf Seite 1von 220

Balancing CoreXL and SecureXL

Michael Endrizzi
Director of Services and Training
mendrizzi@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1
1/24/2015 Midpoint Technology, Inc
Who Is Instructor?
• Michael Endrizzi – Midpoint Tech
• Director of Training and Services
• Age: 56
• CCSA,CCSE, CCSMA, CCISP, ITIL
• Information security since 1982
• Developer on Secure Computing
Sidewinder firewall in 1993
• Worked with NSA
• Owned information security
businesses
• Independent security consultant
for 16 years
• Working with Check Point since
1996. 10 year hiatus into auditing
now back
• Oh yeah, I like rock climbing

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
2
1/24/2015 Midpoint Technology, Inc
Balancing Check Point Systems

• Overview
• Linux Review
• SecureXL
• CoreXL
• Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
3
1/24/2015 Midpoint Technology, Inc
Danger Will Robinson
• This class will teach you how to tune
BUSY/Critical firewalls
• Commands you learn here will take
affect IMMEDIATELY
• These commands are to be used delicately.
• By issuing these commands without fully
testing them, you may negatively impact
performance on CRITICAL firewalls.
• R77.10 introduced different behavior on some
commands. Read the docs.
• Most commands do not allow you do easily
undo themselves.
• You have to use your notes and photo clips to
record the current configuration in case you
wish to revert!
• If you change multiple items at once, you may not
be able to undo the changes in case of failure.
• Backups will not save this information. You
have to do a snapshot/restore in order to recover.
• DISCLAIMER: By reading this line you are totally
responsible for all changes to your environment.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
4
1/24/2015 Midpoint Technology, Inc
What is SecureXL??
SecureXL (aka Secure Network Distributor) is a way of speeding up rule processing by
analyzing traffic patterns and handing off known and previously approved traffic to
high throughput traffic handlers.

VS1
Secure Network Distributor

SND • Processing incoming traffic from the network interfaces


• Securely accelerating authorized packets (if Performance
Pack is running) - SecureXL
• Distributing non-accelerated packets among kernel
instances - CoreXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 5
What is CoreXL?
CoreXL allows you to increase the throughput capacity of your firewall
platform thru the utilization of multi-processors concurrently processing
firewall requests.

• More expensive to purchase


• Cost efficient
and maintain
• Easier to manage
• Easier to debug

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
6
1/24/2015 Midpoint Technology, Inc
What is NGF/VSX?

NGF (Next Generation Firewall) is a


standalone gateway running 1 instance
of the firewall module AND has the ability VSX is a physical chassis that runs multiple instances
to run several threat prevention blades of firewall gateways. Think of VMware that runs firewalls
IPS/AV/AB/Threat-Emulation as guests.

FWK VS0 VS1 VS2

Linux 2.6.18-XXcp
All based on Linux

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 7
What is SPLAT/GAIA?

• Secure Platform (SPLAT) is the commands and kernel modules added to


Linux which transforms the Linux platform into a Check Point firewall
• GAIA was a self-contained command environment created to simplify
administration. Looks like Cisco command environment.

GAIA command shell – Self


GAIA contained shell looks like Cisco
CLI
SPLAT command set added to
Linux command set

SPLAT Linux command set

Linux Kernel

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 8
Goal
NGF/VSX + CoreXL + SecureXL = Tuned System

Describes many topics in these articles. After this


course you will be able to understand these SK’s

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
9
1/24/2015 Midpoint Technology, Inc
Course Take Away Slide
This is the take-away slide, the value behind the whole course. If I bore you to
death with 200+ slides, this slide #190(approx) puts to practice what the whole
course is trying to teach:

Allocate CPUs in the following priority order:

1. Share cache for common data


2. Allocate CPUs to busy Internal interfaces
3. Allocate CPUs to slower less busy External interfaces
4. Allocate CPUs to FW instances
5. Remaining threads are usually idle so distribute evenly
and let kernel find idle processor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
10
1/24/2015 Midpoint Technology, Inc
What Do I Need To Know?
Basic Linux CLI experience, What is operating system, What are processes, What is a
processor, What is a cache, . Basic programming: code, data, variables.
Unix ‘top’

Unix ps

/proc file system

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
11
1/24/2015 Midpoint Technology, Inc
Secure/CoreXL Tuning Areas
5 areas of tuning

3) CP Process
VS1 VS2
2) Fw kernel 4) Linux
Affinity Instance
Process
Affinity
FWD syslogd
logging

1) Interface
Affinity SecureXL 0) Rule Processing
SND Secure Speedup
Network
Distributor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
12
1/24/2015 Midpoint Technology, Inc
Firewall Management
SmartCenter Components

Smart Dashboard
POLICY
User Space Edit Policy

CPD Push Policy FWM


FWD (mgt server (Management
(logging) communication) Server)

Kernel Space

Linux TCP/IP

NIC NIC
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
13
1/24/2015 Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
NGF Components

Smart Center MGT Station


POLICY
User Space FWSSD
FWD (spawns security
(logging) servers like SMTP
filtering)
CPD VPND
CPWD
(mgt server (VPN)
(watchdog for
communication) dead processes)

Kernel Space
SPLAT/GAIA Kernel (fwk)
(Security Enforcement)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


14
1/24/2015 Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Smart
Center MGT
VSX/VS Components
Station

cpd cpd cpd

fwk fwk fwk

fwd fwd fwd

vpnd vpnd vpnd

User mode VS0 VS1 VS2


Kernel mode

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


15
1/24/2015 Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
R75.40 VSX – FWK in User Mode

cpd
• Note: fwk was moved to user mode
fwk
• With large number of VS’s, kernel was
getting too big
fwd

vpnd

User mode VS0

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Our Lab
Eth0: 10.2.1.253/24 Eth1: 172.17.1.2/24

Eth0: 10.2.1.101/24

Eth0 : 172.17.1.111/24
Eth3: 172.17.2.2/24
Eth0:1: 10.2.2.101/24 Eth2: 10.2.2.253/24

Eth0:2 : 172.17.2.111/24

Eth0:2: 10.2.0.101/24 Eth0:1 : 172.17.0.111/24

Eth0: 10.2.1.153/24

Eth1:1 : 172.17.0.1/24
VB: internal/internet
Eth0:1 : 10.2.0.153/24
VB: Host/Host#1

20
1/24/2015
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
20
1/24/2015 Midpoint Technology, Inc
Balancing Check Point Systems

• Overview
• Linux Review
• SecureXL
• CoreXL
• Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
21
1/24/2015 Midpoint Technology, Inc
Linux Kernel Basics

• Linux Overview
• Threads
• Network Processing

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
22
1/24/2015 Midpoint Technology, Inc
Linux History

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
23
1/24/2015 Midpoint Technology, Inc
Evolution of the Kernel
Computer Day 1 Multics-Unix NT - Mach Linux
The User User User
Code
Blob
User User User
User User User

Monolithic Kernel
Kernel Mod Mod
Mod ule
Kernel ule ule
Kernel

• Some tried to simplify the


kernel…. • BACK to monolithic kernel (one
• Programs are • Users have their own process • Kernel broke self into smaller memory space). Minimal context
one compiled resources (memory, files..) processes, some run in user switch and no slow IPC
binary all • Monolithic kernel has its own space • Dynamic added/subtracted modules.
superuser resources (memory, code • Processes used Inter Process • User processes don’t call kernel
• PROBLEM: base). Communication to work process, user processes go into
hacking, bad • Kernel could multiprocess together kernel mode with shared data
code user processes but not self structures! Minimize context switch
corrupted • PROBLEM: SLOW In-efficient and data passing (more later on this)
whole system PROBLEM: too big expensive because of context switches
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
for hardware at time Proprietary and Confidential: No part of this document may be reproduced without permission from
24
1/24/2015 Midpoint Technology, Inc
Linux Differentiators

• Original kernels were monolithic:


• Single binary process
• Single processing thread
• Cooperative Multi-tasking – Could not pre-empt kernel processing
• Single Address space

• Linux
• Linux Tervald – Still heavily involved vs design by committee vs free-for-all
• Pre-emptive kernel – Most kernel tasks can be pre-empted for higher
priority tasks
• Modular – Kernel functions can be dynamically created/removed
Check Point implements firewall subsystem in these modules
• Multi-processor support
• Threads = Processes (Unique to Linux – will explain later)
• Users can see internal kernel data in sysfs file system. Looking glass
into kernel internals

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
25
1/24/2015 Midpoint Technology, Inc
Linux 2.4 to 2.6

Introduced to reduce the time a kernel task held on to the processor locking
out other tasks. Overall increase in efficiency and multi-processing support.

1. Scheduler – Improved fair scheduling with 100’s of processors


2. Threads – Process = thread. No special handling for threads
3. Interrupts – Can be pre-empted and no locking out all CPUs while
processing interrupts
4. Pre-emptive Kernel – The whole kernel is pre-emptive. Can be
interrupted at any point

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
26
1/24/2015 Midpoint Technology, Inc
Linux Kernel Structure
An application process consists of:
• User Identity & Permissions
• Code and Execution Pointer When applications call the kernel,
• Data Scratch Pad (Stack and the kernel does not “take over” as a
Hash) separate entity. Applications
• Check Point VSX Firewall Kernel transform into the “The Hulk”. The
and Helpers application processing thread takes
on kernel permissions, resources
and code base to complete a task. If
the thread does not go to sleep,
processing returns to application
mode (Bruce Banner) almost as if it
made an internal function call.

Most kernel work done


here. Talks to hardware, handles
• Resource allocation hardware interrupts.
• Scheduling In Check Point, drivers are
• Security right from manufacturer.
• Memory Mgt Some appliances have
• File System Mgt modifications
• Communication
• Check Point NGF
FirewallKernel

http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
27
1/24/2015 Midpoint Technology, Inc
Linux Kernel Basics

• Linux Overview
• Threads
• Network Processing

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
28
1/24/2015 Midpoint Technology, Inc
What is a program?
Simple terms: A program is made up of:
1. CPU Instructions – Does the work
2. Data – Scratch pad area
3. Security attributes - restrictions

• CPU Instructions
• Data
• Security attributes

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
29
1/24/2015 Midpoint Technology, Inc
What is a process?
A program is a compiled binary sitting on a disk in a file.
A process is when the program is executed on a processor, assigned memory, and
is managed by the OS kernel.

Program becomes a Process


• CPU Instructions
• Data
• Security attributes

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
30
1/24/2015 Midpoint Technology, Inc
What Does 32-bit User Program
Data look like?
All 32-bit programs have 4 gig
of memory available to them.
This is how it is allocated.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
31
1/24/2015 Midpoint Technology, Inc
Why Care?:
CPU vs. Memory problem

Are you chasing a CPU or memory problem? Need to know how lack of
memory will slow a system and make it feel like it is a CPU problem.

‘top’->’f’->’u’

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
VM Why do you care?

If you ever want to figure out why a process is swapping, you need
to be able to know what parts of a process are taking too much space.

Virtual
Size

Physical
Memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
33
1/24/2015 Midpoint Technology, Inc
Stack Frame
A program is broken into functions. When a function
is called, the program has to save information about
where it was and pass information to the new function.
This placeholder is called a stack frame. Stack frames are To High Memory Stack
like a track of cookie crumbs to help you go back to where
External Environment
where.

Parameters

GROW
Program Counter

Saved Frame Pointer

Local Variables

To 0 memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
35
1/24/2015 Midpoint Technology, Inc
Heap space and Memory Leaks
Programs sometimes need dynamically
allocated and sized chunks of memory for maybe network
packets. The program uses the malloc call to get that
memory. It came off the heap space.

If the programmer forgets to free the memory, then you have


a memory leak.

Sometimes heap space becomes so fragmented with mixed


free and allocated memory, that programs slow down
because its hard for them to claim and release memory
efficiently. Thus ‘reboot!!!’.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
36
1/24/2015 Midpoint Technology, Inc
Buffer Overflow

Stack Code
Length: 1000 bytes
Length: 256 bytes
301 LOGIN:
+ username +

Packets
Mail Fragmented
Program Running as Super-User

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
IMAPD Attack
Authentication Attempt Backfires
To 0 memory Stack Attack
Login
Local Variables String

Mike
UserName[256] Password

Saved Frame Pointer

Program Counter New PC


EndOfBuffer
Parameters
NULL

Environment

To High Memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


38
Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Virtual Memory
4GB Process A 4GB Process B 4GB Process C 4GB Process D

NOTE: Most programs don’t use all 4 gig at


once. Only use small portion.
How much total memory is needed
to store all these processes - MAX???
NOTE: This is for 32-bit systems. 64-bit can
see 264 = 16 exibytes of memory
Not needed right now

+
Virtual memory is 3 things: so swap out
1) Allows processes to think they
own all memory
10GB Disk Swap space
2) Allows processes to ignore
physical memory limitations Assuming all processes needed 3GB User space
We would need at a minimum
3) Paging system: That uses disk 12GB (3GB+3GB+3GB+3GB)User + 1GB kernel = 13GB
to swap out sleeping data to Physical and Swap space to hold all these programs.
temp storage area to free up
physical memory for active
processes.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Demo: Using VIRT Memory

Here in my ‘memgrow’ program I allocated 1GB of HEAP memory, but didn’t


read/write to it. Notice how the kernel allocated the VM space to my process but
did not actually map it into physical memory or SWAP it out because we are
running out of space (I only have 1 gig of physical memory)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
DEMO: Writing to VIRT Memory
Here the same ‘memgrow’ program BUT I wrote to every byte so the kernel had
to map the page to my process and bring it into physical memory. And when physical
memory fills up…then swap it out…
WHAT IS VIRTUAL MEMORY????
1) Process thinks it owns all 32-bit 3GB by itself
2) Process doesn’t understand physical memory constraints of 1GB (only 32/64
max addressable memory
3) Paging swapped out data that the kernel could not keep in physical memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
User Program Security Attributes
Computers Day 1 allowed a program to see and do everything. But programs
started to overrun each other. So Day 2 a kernel was developed that was the
traffic cop between programs so they wouldn’t bump into each other OR
corrupt the kernel itself.

NOTE: On many embedded systems there is no kernel/user/security separation.


All processes can see and do everything. For example: SCADA controllers that run
machinery. Programs that monitor the brakes in your car.

User Level
• CPU Instructions • CPU Instructions
• Data • Data
• Security attributes • Security attributes

Kernel Level

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
43
1/24/2015 Midpoint Technology, Inc
What is the Kernel?
The ‘kernel’ is responsible for resource management.

User Level

• Access to physical devices


• Security separation
• Arbitrator of shared resources – who gets what first
• Virtual memory management
• Clock ticks and time slices
• Process management
• Scheduling processes
• File system access
• Network routing
Kernel Level

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
User Program Entering The Kernel
Linux TODAY: When a process makes a system call, the process does not hand over processing to
another entity. The processing thread gains access to kernel resources
(after appropriate security checks) so the kernel is actually executing “on behalf
of a specific process”. The process running in kernel mode now has access to both
user and kernel memory. Its like a program making a function call and
the function has enhanced magic powers, and then returns to the main user process.
Just like “The Hulk”
User Space

Kernel Space SYSTEM CALL


(security checks)

+
• Kernel Data Structures
• Superuser privileges

My friend Arah becoming the HULK

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com 45


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc
3 Processing Contexts
In Linux a processor can be in 1 of 3 processing contexts at any time….period

1. In user-space, executing user code in a user


process
2. In kernel-space, in process context,
executing on behalf of a specific user
process
3. In kernel-space, in interrupt context, not
associated with a process, handling an
interrupt
Why do you care?

http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
46
1/24/2015 Midpoint Technology, Inc
Pre-Emption and Context Switch
In order to avoid one process from dominating a processor pre-emptive multi-tasking was
introduced. This is where one process B can interrupt another process A at any time (pre-
empt) resulting in a context switch to the process B.
Prior to mid-1990’s context switches (stopping one process and starting another)
were CPU/Memory intensive. Kernels had to copy the process state into internal kernel
Process Control Blocks (PCBs) to save the state and then copy out a PCB to the
new process.
Process A Process B

• Virtual Memory Space


• Stack and Hash Tables Context Switch • Virtual Memory Space
(Scratch Pad) • Stack and Hash Tables
• CPU Instructions (Scratch Pad)
• Processor registers • CPU Instructions
• Security attributes • Processor registers
• Security attributes

Kernel

Process Control Block Process Control Block


© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
47
1/24/2015 Midpoint Technology, Inc
Old Interprocess Communication
Copy Data
In the old days communication between processes and to/from the kernel data
was copied. Very inefficient.

Process A Process B

Copy this to Process B

Kernel

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
48
1/24/2015 Midpoint Technology, Inc
New Interprocess Communication
Share Data
So a shared memory mechanism was developed using virtual memory tricks
that allowed memory spaces to be shared between processes and the kernel.

Process A Process B

You can see some of my data

Kernel

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
49
1/24/2015 Midpoint Technology, Inc
What is a Multi-Processor
Multi-processors come in several different varieties
• Multi-Processor: Multiple physical CPUs in a single chassis
• Multi-Core : Multiple processor cores on a single physical CPU
• HyperThreading: Simulated multiple processors with threads on a single physical core
• Combination of the above
• No matter which of the above configurations, the OS sees logical processors. The implementation is opaque.
• Note where the caches are! Remember this when you assign interfaces so you keep the cache hot

OS sees 4
‘logical’
processors

Logical
Processors

Threads

Cores

Physical
CPU http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
50
1/24/2015 Midpoint Technology, Inc
What are CPU Threads
Enabling Symmetric Multi-Threading (SMT) or HyperThreading (HT) doubles the number of logical processors.

• Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting
pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel
processing.
• Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time.
• With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently.
So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time
slices.
• Performance improvement 30%??? on CPU intensive items. I/O intensive theoretically could slower.
• Shares a cache

Without Hyperthreading, Managed by Kernel


Thread 1 Thread 2 Thread 1 Thread 2

WithHyperthreading, Managed by HW

TIME
http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
51
1/24/2015 Midpoint Technology, Inc
Example:HP-DL380p Multi-Core
No HyperThreading
HP DL380p –
CACHE
2 Physical Processors
8 Logical Processors
CPU0
CPU1
CACHE

OS sees 8
HP DL380p – 8 Logical Processors ‘logical’
processors

No Hyper
Logical
Threading Processors

Threads
Cache
Cores

Physical
CPU
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
52
1/24/2015 Midpoint Technology, Inc
Example: CheckPoint 12600
12 core

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
53
1/24/2015 Midpoint Technology, Inc
What is a multi-processing?
Day 3 – As loads increased computer designers decided to add multiple processors to the system.
Processes lent themselves nicely to the paradigm, each process could float to a free processor and
execute.

Process A Process B

• Virtual Memory Space


• Stack and Hash Tables • Virtual Memory Space
(Scratch Pad) • Stack and Hash Tables
• CPU Instructions (Scratch Pad)
• Processor registers • CPU Instructions
• Security attributes • Processor registers
• Security attributes

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
54
1/24/2015 Midpoint Technology, Inc
Shared Data Contention
LOCKUP!!
The biggest problem with concurrency is access to shared data. If not programmed
correctly two processes can fight over updating a shared piece of data and they
fight to the death. What you see is a frozen monitor!!!

• Virtual Memory Space • Virtual Memory Space


• Stack and Hash Tables • Stack and Hash Tables
(Scratch Pad) (Scratch Pad)
• CPU Instructions • CPU Instructions
• Processor registers • Processor registers
• Security attributes • Security attributes NO! I’m going to write to
I’m going to write to variable X first!!!
variable X first!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Threads To The Rescue
If a process wanted to enable concurrent processing (web server serving up pages, word
processor with multiple open documents), running multiple processes was inefficient.

Threads were created to support concurrent processing using SHARED data and NOT copying
data between processes or resource heavy context switching.

SHARED DATA
• Virtual Memory Space
• CPU Instructions
• Processor state
• Kernel threads
• Security attributes
• File system
Thread C
• Signals
Thread B
Thread A

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
56
1/24/2015 Midpoint Technology, Inc
Light Weight/Quick Context Switches

Thread context switches are very fast because the kernel only has to save a several
registers (TCB) (approx 96 bytes vs. approx 1K+ with heavy process switches). This is
because all the non-saved data is shared between the threads so its live data and no
need to save it and restore it. In addition, threads can see into each others address space
because remember everything in the process is shared.

Thread A Thread B
• Thread registers Mini- Context Switch
• Program counter
• Stack pointer

Kernel

Thread Control Block(TCB) Thread Control Block(TCB)


https://courses.cs.washington.edu/courses/cse451/11sp/section/kim_section4.pdf
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
57
1/24/2015 Midpoint Technology, Inc
Threads and Multi-Processing
Threads naturally lend themselves to multi-processing…you can concurrently
run several threads on different processors.
Of course there are shared data contentions that must be dealt internally by
the threads

SHARED DATA
• Virtual Memory Space
• CPU Instructions
• Processor state
• Kernel threads
• Security attributes
• File system
Thread C
• Signals
Thread B
Thread A

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
58
1/24/2015 Midpoint Technology, Inc
Threads and Scheduling
Linux 2.6.18 changed its default scheduling to favor low latency processing for more real-
time applications. It is called Completely Fair Scheduling (CSF). Its simple, when the kernel
schedules a process it chooses a process that has used the least amount of its time slice.
That way CPU-lite processes are favored. These are usually ones that are blocking waiting
for input like network interrupts, keyboard, graphics, etc.
When a process is ready to run, the kernel looks for:
1. What CPU is free on the affinity list for the process.
2. Did it run on the CPU before so the cache is fresh with my data?
3. Am I still in default mode where no autobalance occurred and so choose CPU 0
(mostly interrupts)

Thread B Thread C
Which CPU?
Thread A
Which CPU?

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
59
1/24/2015 Midpoint Technology, Inc
Processor Affinity
If you have a thread that is usually very busy, you can wire it to a single CPU (or subset of CPUs).
That way all the data is cached locally on the single CPU and it can usually run less interrupted on that
CPU. This is called Processor Affinity. A process/thread has an ‘affinity’ for a CPU. Can be done in Unix
and Windows Oss Check Point leverages this in their CoreXL technology.

• Virtual Memory Space


• CPU Instructions
• Processor state
• Kernel threads
• Security attributes
• File system Thread C
• Signals
Thread B
Thread A

CPU1 CPU2 CPU3


© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
60
1/24/2015 Midpoint Technology, Inc
Types of Threads
3 types of threads:
1. User threads: Totally internal to process, kernel cannot see them limited multi-processing
capabilities for example they couldn’t block to sleep or I/O.
2. User threads mapped onto kernel threads. Full multi-processing capabilities for user process
3. Internal kernel threads: Used only by kernel for internal kernel processing not visible
to user processes
http://linuxgazette.net/23/flower/threads.html
http://www.thegeekstuff.com/2012/03/linux-threads-intro/

User Level Internal kernel threads


• Virtual Memory Space
• CPU Instructions
• Processor state
• Kernel threads
• Security attributes

Kernel Level User Threads mapped to Kernel Threads

Internal Kernel Threads


• Scheduler
• Blocking I/O handling
• Thread table
• Monitoring
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
61
1/24/2015 Midpoint Technology, Inc
Thread Demo
Internal kernel threads have a [] around them. User threads, usually do NOT have [] on
them. NOTE: sometimes user processes do, but its because the kernel can’t find its
command parameters and notates it with [] which is confusing. Other commands can
filter this out.
“ps –ef”

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
62
1/24/2015 Midpoint Technology, Inc
Thread Demo: Windows Threads
Even windows has threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
63
1/24/2015 Midpoint Technology, Inc
Demo: Kernel Thread
Kernel threads have no virtual memory size because they all share
the VM of the kernel and no allocated extra memory like user threads can
be allocated.
Remember this: will be similar when we look at CP NGX

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
64
1/24/2015 Midpoint Technology, Inc
Thread Demo: Multi-Processing Threads

You can watch threads switching between processors to see which processor is busy

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
65
1/24/2015 Midpoint Technology, Inc
Linux Unique Thread Architecture
Other OS’s have a concept of Light Weight Processes (LWP) Linux 2.6 everything is a thread, and they are self
(threads) where the parent process spawns owns, and managed. They keep track of their own state and don’t
manages the threads. rely upon a parent to manage them. Linux CAN
simulate LWP to the outside world and this is what you
Think mamma black bear protecting her cubs will see in the demo/labs.

Think busy bees – once born


they are off on their own

Main Process Parent Code

Thread A Thread B Thread C

Thread A Thread B Thread C

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
66
1/24/2015 Midpoint Technology, Inc
Linux Process to Thread Relationship

So a Linux process is like a beehive that keeps all the common data/honey. The threads
are like the bees that go off and do their work and bring back data/honey where it is all
shared between them.

• Virtual Memory Space


• CPU Instructions
• Processor state
• Kernel threads
• Security attributes
• File system
• Signals

Thread A Thread B Thread C

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
67
1/24/2015 Midpoint Technology, Inc
Clone() – Threads and Sharing
This system call is the heart of threads and sharing. Depending on what the programmer
tells the thread to share, the thread could act like a full blown HEAVY process (old days),
or like a little lite worker bee that doesn’t carry any baggage with them (Linux today). Other
Unix’s are starting to implement this in some fashion but Linux was first.
int __clone(int (*fn) (void *arg), void *child_stack, int flags, void *args)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
68
1/24/2015 Midpoint Technology, Inc
Thread Summary

• Threads can have 3 contexts:


• User
• Kernel
• Interrupt (very special type of thread, very limited)
• Threads are the workers and the data is ‘usually’ stored in a shared common
process space
• Threads enter kernel space and take on kernel context: virtual memory space,
and security context.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
69
1/24/2015 Midpoint Technology, Inc
Linux Kernel Basics

• Linux Overview
• Threads
• Network Processing

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
70
1/24/2015 Midpoint Technology, Inc
Interrupts
The kernel interacts with external devices such as network cards in two ways:
1. Interrupts – Only interrupt the kernel when there is something to do
can be more efficient
2. Polling – Continuously see if there is any activity on the device. Very
inefficient and slows down the kernel
Modern Linux device drivers use a combination of both (New API - NAPI). They wait for an interrupt to
occur and begin processing. They will then disable the interrupt and go into a
polling mode until there is nothing more to do (no more data). They will then
re-enable the interrupt to be alerted for more activity.

Interrupt Handler
Kernel
Poll

Hey WAKE UP

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
71
1/24/2015 Midpoint Technology, Inc
Interrupt Priority
Interrupts have a priority system. Interrupts such as the system clock
have to be processed right away and are a higher priority. They will stop
all processor activity (even lower level interrupt handlers) to be handled.

Lower priority interrupts will disable its own interrupt line (eg. Network card)
so that it doesn’t get interrupted from the same source and run uninterrupted
(unless a higher priority interrupt comes along).

Kernel

LOW PRIORITY
HIGH PRIORITY

System Clock

http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 493 NAPI

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
72
1/24/2015 Midpoint Technology, Inc
Advanced Programmable Interrupt Controller (APIC)

APICs are the hardware that manage interrupts. A motherboard had one I/O APIC that
interfaces with the hardware and talks to LOCAL APIC controllers embedded within the
CPU.

In SMP environments where IRQs can be handled by multiple CPUs APICs can be
dynamically programmed by the kernel to direct IRQs to a specific CPU for balancing out
the handling of IRQs from external devices.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
73
1/24/2015 Midpoint Technology, Inc
Top/Bottom Half Interrupts

Interrupt processing is split into 2 parts:


1. Hardware Interrupt (Top): Device specific, stops processor, has to be quick
2. Software Interrupt (Bottom): Thread-like, heavy lifting when CPU has time

Software Iinterrupt (Bottom ½)


Heavy lifting

Stop processor (Top ½)


Quick and get out

Schedule SW Interrupt: When processor has time


do generic processing of packets.

HW Interrupt: Run device specific handler

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
74
1/24/2015 Midpoint Technology, Inc
‘top’ view of interrupts

Hardware interrupt handles Software interrupt handles


device drivers
‘top’ – ‘1’ for CPUs

http://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468/ref=sr_sp-atf_title_1_1?s=books&ie=UTF8&qid=1389542475&sr=1-
1&keywords=linux+kernel+development Page: 3322
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
75
1/24/2015 Midpoint Technology, Inc
Interrupts are not Process/Threads
This is why you can’t see interrupts in a ‘ps’ or a ‘top’, different data structure than
processes/threads. Very much like them but CANNOT SLEEP/BLOCK!

Soft Interrupt Table array


Softirq[0] Softirq[1] ……… Softirq[31] Process/Thread Table
(used for ‘ps’ and ‘top’)
Hard Interrupt Table linked list

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
76
1/24/2015 Midpoint Technology, Inc
Device Drivers
A network device driver runs within the kernel and has two primary functions:

User Process

Kernel
Driver Transmit Functions Interrupt Handler

Device Driver

1) Organize data so 2) DMA data to kernel


NIC can grab it memory and send
interrupt when ready
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
77
1/24/2015 Midpoint Technology, Inc
DMA vs CPU copy
Note that a Direct Memory Access (DMA) processor is responsible for transferring
data between the NIC and the host. This allows the CPU to parallel process other
activities while the transfer occurs. There my be some bus contention between
the CPU and DMA, but not as bad as if the CPU had to perform the transfer.

User Process
Kernel

DMA Interface

Host
Initiate Data DMA chip
Transfer!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
78
1/24/2015 Midpoint Technology, Inc
Packet Journey – User to Device

Linux Kernel Net I/O Kernel Thread Linux


On Behalf Of User
Call to device driver to put Copies into DMA space and
User
Send device interrupt
data into the right massages data through Space
structures for the NIC and TCP/IP stack
when data is ready to be tell the NIC it can copy.
copied by DMA transfer

send(socket, msg, strlen(msg), 0)


hard_start_xmit()
Application

eth0
TCP/IP Stack

DMA
Space

http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview
http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505
http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
79
1/24/2015 Midpoint Technology, Inc
Packet Journey – Device To User
Linux Kernel Linux User Space
Software Iinterrupt (Bottom ½) User process
Hardware User thread
Massage packet through waits for data on
Interrupt (Top ½) Inside kernel
TCP/IP Processing socket
(Stops processor) context

int recv(int s, void *buf, size_t len, int flags);

Signal to continue Application


DMA
Space User
eth0 Space
TCP/IP Stack
IRQ 177

Soft IRQ copies


data to user
space

Big job! Schedule


Device DMAs software
data to kernel interrupt
DMA memory

http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview
http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505
http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
80
1/24/2015 Midpoint Technology, Inc
Monitor # of Interrupts per Device
/proc/interrupts keeps track of # of hw interrupts per interface since boot.

Linux will use eth0 as default for network cards until system gets busy then it tries and
re-balance between CPUs (see eth0)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
81
1/24/2015 Midpoint Technology, Inc
Interrupt Affinity
In Linux, you are able to wire an interrupt to a specific CPU. This is called
Interrupt Affinity. Once again this allows data to be cached locally on a single (set)
CPU for the interrupt handler.

Linux will start with CPU 0 handling all interrupts. (?? Does Linux auto balance??)

Interrupt affinity is used by Check Point CoreXL as we will see in the next section.

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-cpu-irq.html

IRQ 32

CPU 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
82
1/24/2015 Midpoint Technology, Inc
Linux Network Summary

• Interrupts come in 2 halves


• Top – Hardware device driver – Quick and get out
• Bottom – Heavy lifting of data through kernel
• Interrupts are NOT threads or processes. They have many things
in common but you can’t view them and are internal to the kernel.
• Device drivers come in 2 halves
• Transmit – send data to device
• Receive – Usually interrupt driven to pick up packets

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
83
1/24/2015 Midpoint Technology, Inc
Linux Evolution wrt
SPLAT/GAIA
Date Linux Release CP
Release
2000 Linux 2.2 R 4.0
Nov 2001 Linux 2.4 NG FP1
Jan 2008 Linux 2.6 R65 HF02
Today Linux 2.6-18cp R77.10

http://en.wikipedia.org/wiki/Check_Point_VPN-1

Linux Tidbits:
GPL (GNU Public License) –
• No custom mods to Linux, they have to be shared with Linux
community as source code. There is a version 2.6.18cp (not sure status)
• Can compile code with GNU compiler and keep source
private. CP uses internal kernel modules to do this.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
84
1/24/2015 Midpoint Technology, Inc
Balancing Check Point Systems

• Overview
• Linux Review
• SecureXL
• CoreXL
• Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
85
1/24/2015 Midpoint Technology, Inc
Why SecureXL??
The classic problem has been at noon when everyone having lunch at their
desk, everyone starts browsing the Internet and thus slowing down business traffic.
In addition as more applications become web-oriented HTTP traffic is
dominating the network.

SecureXL was primarily created to address the web-browsing-at-noon-problem.

Symptom Verify

Highly utilized system ‘top’, ps –o psr,command

High network traffic on 2 interfaces /proc/interfaces

High HTTP traffic SmartLog

Ifconfig is dropping packets, retransmits Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
86
1/24/2015 Midpoint Technology, Inc
HTTP 1.0

HTTP 1.0 uses multiple concurrent requests to retrieve a multi-part web


page for the user.

Text

Picture

Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
87
1/24/2015 Midpoint Technology, Inc
HTTP 1.1

HTTP 1.1 uses a single request to retrieve a multi-part web


page for the user.

Text

Picture

Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
88
1/24/2015 Midpoint Technology, Inc
HTTP processing

SecureXL is smart enough to know that all packets after packet #1


are all going to the same site and will hit the same rule.
So SecureXL speeds up the processing of these packets by bypassing
the full rule processing and just relying on state tables built by packet #1
to send the packets through the firewall.

Text

Picture

Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
89
1/24/2015 Midpoint Technology, Inc
What does SecureXL accelerate?

Secure XL can accelerate (bypass full rule processing) in two


instances:
1) Subsequent connections to the same dest and port
2) On a single connection, packets 2 thru N.

Text

Picture

Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
90
1/24/2015 Midpoint Technology, Inc
NGF Process Mapping
SND does the acceleration and packet distribution. NGF has 1 SecureXL module to
accelerate network packets. Firewall dispatcher sends packets to the right firewall
instance (there can be many..next section)

https://downloads.Check Point.com/fileserver/SOURCE/direct/ID/7513/FILE/CoreXL_Advanced_Configuration_Guide.pdf

User mode

fwk
Kernel mode

Firewall Dispatcher(fwkdrvr)
SND Performance Pack Packet Handler
(SecureXL acceleration)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
91
1/24/2015 Midpoint Technology, Inc
VSX Process Mapping
In VSX, SND is still one module but SND is aware of individual VSs. SecureXL can
be turned off/on per VS but SND shares info about all VSs when making
acceleration decisions for the whole chassis.

When Check Point moved the fw kernel from the Linux kernel to User Mode, they
left only a little bit of code to work with the firewall dispatcher in place. Other than
that it was a clean compile of the User mode kernel…This was not a massive rewrite

VS0 VS1 VS2

User mode
Kernel mode
Firewall Dispatcher (fwkdrvr)
SND
VS0 SecureXL VS1 SecureXL VS2 SecureXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
92
1/24/2015 Midpoint Technology, Inc
NGF Firewall Kernel Modules
Here you can see the kernel modules for NGF vs VSX. Basically the same firewall module
is used for both. But…probably…VSX only uses the bottom half processing because the
firewall itself is in user space.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
93
1/24/2015 Midpoint Technology, Inc
SecureXL Acceleration
There use to be a hardware acceleration device by Nokia, now its in software. CP
recently put it back in hardware

SecureXL acceleration
1) Subsequent packets from a
single connection
2) Subsequent packets from
the same source IP, same dest
IP and same dest port
(multiple HTTP requests to
same dest)
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
94
1/24/2015 Midpoint Technology, Inc
Packet Journey Thru SecureXL
Linux Kernel Linux User Space
Concurrent processing
of SI’s is possible unlike
Device driver hogs HW interrupts. Can be
Hardware processor. Can’t be interrupted.
Core 3
FW
Interrupt interrupted. Just Instance 3
(stops whole processor transfer data.
single thread)

SND
Device FW FW
Interface
Driver Instance 1 Instance 2
eth0

eth0 Software Core 0 Signal Core 1 Core 2


Core 0
Interrupt ‘Continue’
IRQ 177

Eth0 needs
service
Can’t accelerate
Big job! Schedule this, send on to a
software specific FW SND picks instance
interrupt Instance 1 to process packet

Standard Linux Processing Core XL

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
95
1/24/2015 Midpoint Technology, Inc
Firewall Chain – SecureXL Modules
Once packets make it through SecureXL, these are the modules in the
firewall chain that build the SecureXL connection tables and sync with
the SecureXL module itself.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
96
1/24/2015 Midpoint Technology, Inc
SecureXL and IPS/AV/Bot Integration
When IPS/AV/Bot is enabled, not all traffic can be accelerated because it has to be inspected
by the IPS engine.

PSL is the Packet Streaming Library for re-assembly


of IP packets so the IPS/AV/Bot can look for signatures.

SecureXL can forward packets directly to the PSL


and bypass firewall processing. This is called
‘Medium Path’ because it bypasses rule checking on
the 2nd+ packets

Fwaccel stats – SecureXL statistics for Medium Path

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
97
1/24/2015 Midpoint Technology, Inc
VSX Detailed Internals
Firewall in user
Packet stays in pool VS2 mode inspects and
Should I creates pass/fail
Accelerate? VS1
Packet Data
HW Interrupt Pool VS0
Construct The Packet
Packet Meta-Data Pointer to packet
SND Queue is what moves
fwk
Outbound to another VS

Fwkdrv in Fwkdrv out


Write pass/drop to msg q
Accelerate

F2F F2F
Outbound to another VS

Dispatches to right IP Stack


VS
Inbound
Message Queue
SND Fwdrv deq
Outbound Legend
Implements action
from fwk to Kernel Code
Device Driver pass/drop
User Space Code

Shared Memory
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
98
1/24/2015 Midpoint Technology, Inc
SecureXL Demo - VSX

• Pass 2 gig file thru a firewall


• R75.40VS VSX
• Traffic going through fw-vsx1 a virtual firewall
• All run inside Virtual Box
• No other traffic

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
99
1/24/2015 Midpoint Technology, Inc
VSX – SecureXL On
SecureXL runs as a software interrupt within the Linux kernel. You can
see the %si get higher when it is busy.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
100
1/24/2015 Midpoint Technology, Inc
VSX – SecureXL off
When SecureXL is OFF, you can see the fwk1_dev thread handle the
work.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
101
1/24/2015 Midpoint Technology, Inc
AutoBalance Interfaces
Note how the interrupt handling got shifted from CP0 to CP1. More on this later

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
102
1/24/2015 Midpoint Technology, Inc
SecureXL Demo - NGF

• Pass 2 gig file thru a firewall


• R75.40VS NGF Standalone gateway
• All run inside Virtual Box
• No other traffic

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
103
1/24/2015 Midpoint Technology, Inc
SecureXL OFF – FW busy
Worker threads are idle, SND and FW sharing the work
Busy kernel threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
104
1/24/2015 Midpoint Technology, Inc
SecureXL OFF – FW sees packets
Firewall sees all packets in fw monitor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
105
1/24/2015 Midpoint Technology, Inc
SecureXL ON– FW idle
Worker threads are idle, SND is doing all the work
Work done in
Work done in SND
SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
106
1/24/2015 Midpoint Technology, Inc
SecureXL ON– FW no packets
The first packet may/not be seen if its in the state table already. But no others

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
107
1/24/2015 Midpoint Technology, Inc
4 types of acceleration

1. Standard state table We talk about these


2. Accept connection templates (e.g. like HTTP)
3. NAT templates – Perform NAT in SecureXL and not in the firewall kernel (sk71200)

[Expert@HostName]# echo 'cphwd_nat_templates_support=1' >> $FWDIR/boot/modules/fwkern.conf


[Expert@HostName]# echo 'cphwd_nat_templates_enabled=1' >> $FWDIR/boot/modules/fwkern.conf
[Expert@HostName]# cat $FWDIR/boot/modules/fwkern.conf
[Expert@HostName]# reboot

4. Drop templates (sk66402)(R76- HF/R76+Standard) – Drop packets in SecureXL, negation of


rules.

5. Other drops (have to test) SAM, DDOS, SmartEvent/IPS triggered.


CONFLICT: sk33781, sk66402, sk98348 all conflict with one another on this topic. Have to test.
Sk98348 says drop templates are the negation of the security policy and drops are accelerated if it does not
match a rule.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
108
1/24/2015 Midpoint Technology, Inc
Standard State Table
Standard state table tuples will be accelerated. Here you
can see a state table entry. Subsequent packets will get
accelerated.
fwaccel conns

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
109
1/24/2015 Midpoint Technology, Inc
Accept Connection Temples
Accept connection templates that will be accelerated. Here you can see
where subsequent connections from the same source IP will be accelerated.
We created multiple SSH
Sessions through the firewall same
Client->Server Our demo had 1
accept template

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
110
1/24/2015 Midpoint Technology, Inc
fwaccel stat
This command provides the status of SecureXL

State table

Accept
connections

DOS
drops

NAT in SecureXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 111
fwaccel stats

This commands provides you statistics on which packets are accelerated


and which are forwarded To Firewall(F2F) for rule processing.

Total acceleration
“C” Current Counts
# from accept
templates
NAT performed by
SecureXL
PXL: PSL + SecureXL
IPS packets

Connections sent to
firewall, NOT
XL/SLOW PATH

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
112
1/24/2015 Midpoint Technology, Inc
Debug Tips

• If you have any random issues, immediately turn off SecureXL to determine
if there is a difference
• Using ‘top’ to monitor performance, turn SecureXL on/off and see what %SI is
doing
• Might have to distribute SecureXL across multiple cores if %SI is busy and doesn’t
autobalance. See next section.
• Monitor stats to make sure both state table and connection templates are being used
• Move HTTP 1.0 type protocols to the top of the rulebase so they get hit
• Avoid protocols that disable connection template acceleration (more on this at end)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
113
1/24/2015 Midpoint Technology, Inc
SecureXL Licensing

• ADN – Advanced Data Networking & Clustering (formerly ACCL)


• SecureXL
• Dynamic Routing
• CoreXL????????
• ClusterXL
• QoS,load balancing, ISP redundancy
• Looks like CPSB-ADNC
• Platform
• Appliances – All inclusive
• Open Platform
• Ala Carte List - $1500

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 114
SecureXL Summary

• Definitely required for high usage gateways


• Easy to administrate (on/off)
• Understand the difference between state table and connection templates
• All takes place in the %SI under ‘top’

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 115
Balancing Check Point Systems

• Overview
• Linux Review
• SecureXL
• CoreXL
• Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
116
1/24/2015 Midpoint Technology, Inc
Why CoreXL
CoreXL allows you to utilize multiple processors on a single chassis for
concurrent processing of firewall requests in order to expand capacity
and reduce latency on your existing platform.
Cheaper to expand capacity on a single bigger chassis than to cluster
multiple smaller chassis (ClusterXL)
• ClusterXL: More expensive to purchase
and maintain
• Cost efficient
• Easier to manage
• Easier to debug

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
117
1/24/2015 Midpoint Technology, Inc
CoreXL Manages 4 Affinity Types

3) CP Process
VS1 VS2
2) Fw kernel 4) Linux
Affinity Instance
Process
Affinity
FWD syslogd
logging

1) Interface SND
Secure
Affinity
Network
Distributor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
118
1/24/2015 Midpoint Technology, Inc
CoreXL

• Interface Affinity
• Instance Affinity
• Process Affinity
• Linux Process Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
119
1/24/2015 Midpoint Technology, Inc
The LIE

• Interface affinity is grey zone. Could be included in both SecureXL and CoreXL
• Interface affinity can be used with SecureXL license and no CORExl license
• Interface affinity can also be used without CoreXL or SecureXL license,
it is a Linux function
• Here - Interface affinity is grouped with CoreXL for completeness and
topic flow

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
120
1/24/2015 Midpoint Technology, Inc
CoreXL Interface Affinity
Interface ‘Affinity’ is the mapping of processors to interfaces to handle network packets.
Default is ALL, which in reality is Core 0.

SND is responsible for managing interfaces assigned to that core. If there are multiple
CPUs handling different interfaces, then each CPU has a different SND.

1) Interface Secure Network Distributor


Affinity SND
• Process network traffic
• Accelerate
• Distribute to firewall
instances

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
121
1/24/2015 Midpoint Technology, Inc
MultiQueue (sk80940)

MQ enables assigning multiple interrupts per interface. Certain interface cards have
multiple TxRx queues per interface. Src/Dst flows are tied to a queue. Then queues
are assigned IRQs and tied to specific processors. This technique optimizes CPU
cache utilization.

https://greenhost.nl/2013/04/10/multi-
queue-network-interfaces-with-smp-on-linux/

Standard on R76 and R77


Previous 71.50

IRQ 2 IRQ 3
IRQ 1 IRQ 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
122
1/24/2015 Midpoint Technology, Inc
MultiQ Restrictions

• R77.10
• Only on appliances…needs the right hardware and drivers
• Supports increased throughput, not so much increased number of sessions
• Based on src/dst assigned to a CPU. So a single high throughput src/dst will
only use 1 CPU and not take advantage of multiple CPUs.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
123
1/24/2015 Midpoint Technology, Inc
Display Interface Affinities

In this complex environment, SND can


concurrently run on cores 0-7

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
124
1/24/2015 Midpoint Technology, Inc
Monitor Hits on SND
/proc/interrupts keeps track of # of interrupts per interface since boot. We can use it to monitor
invokations of SND

In this simple environment SND has ability to concurrently run on any core (that isn’t running
a fw instance), but by Linux default it chooses to run on CPU 0. (Probably not good because
all interfaces and processes will use CPU 0).

So if you have interfaces that are dropping packets, you might want to check this if CPU 0
is busy. POINT: Even though the configuration seems balanced, you need to verify!

SND on core 0 is doing ALL the work


for all interfaces

SND has never autobalanced,


all on Core 0
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
126
1/24/2015 Midpoint Technology, Inc
IRQ to CPU mapping

Linux maps IRQs to CPUs in the


proc/irq directory in the smp_affinity
file

These are used to program the APIC


(interrupt controller chip). By default
a IRQ can run on any processor but
Linux chooses Core 0

Why core 0? Well the kernel doesn’t


want to have the interrupt context
float between CPUs. This way it keeps
interface interrupt data in local cache.

R77.10 changed from ‘all’ to CPU 0.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
127
1/24/2015 Midpoint Technology, Inc
Setting affinity command

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
128
1/24/2015 Midpoint Technology, Inc
Setting interface affinity
sim affinity –s sets the interface affinity by changing the values in the smp_affinity
file. This in turns programs the APIC to send interrupts to a different processor.

NOTE: Interface settings will survive reboot (BUT not CoreXL settings (next)).

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
129
1/24/2015 Midpoint Technology, Inc
Impact Immediate

No reboot, sim affinity –s takes effect immediately

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
130
1/24/2015 Midpoint Technology, Inc
Distribute SND processing
This is how you distribute SND
processing on interfaces that are overloaded and
dropping packets. Give those interfaces/SND VS1
their own processor that does not have other FW
components on it
VS2

SND SND

Busy SND Secure


Network
Busy Distributor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
131
1/24/2015 Midpoint Technology, Inc
SND Chooses non-fwk CPUs

When an interface is set to All, it will attempt to use a CPU that is NOT
being used by a firewall instance. But it will try to use a CPU that is being
used by another interface…in order to keep the the local CPU cache fresh.

So when under low CPU usage, most interfaces will default to ALL (below) and
be autobalanced as CPU and interface activity picks up. The default for ALL
is CPU 0…until CPU activity picks up

R77, interfaces will be set to the default CPU0

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
132
1/24/2015 Midpoint Technology, Inc
Interface Auto - Balance
Look what happens when straining a CPU with a 3 gig SCP transfer
between members directly connected.

Every 60 seconds, CoreXL examines the CPUs to see if they are busy.
If they are busy it will rebalance interfaces to non-busy CPUs.
(fwkernel and Linux processes rebalance every 1-2 seconds).
(NOTE: I do NOT know how to set back to autobalance once you hard set the interfaces
except by reboot on NGF, or factor defaults on VSX)

Below the fw kernel saw the CPU going to 80 si% and rebalanced the interfaces
from ‘all’ to ‘eth1:1’, gave eth1 its own CPU

Before
After

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
133
1/24/2015 Midpoint Technology, Inc
Busy box?
How to tell if a box is ‘busy’??? Here is an example of an Internet gateway
with 10gig interfaces. This is the /proc/interrupts table. You can see that
only CPU 0 has been used to process network packets.

This tells us that the CPUs have not become busy since reboot and the fw kernel
has not done any rebalancing. If the ifconfig shows packet drops, then you have
different issues than CPU not being able to handle the load.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
134
1/24/2015 Midpoint Technology, Inc
Set Interface Affinity – No Performance
Pack

$FWDIR/conf

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
135
1/24/2015 Midpoint Technology, Inc
CoreXL

• Interface Affinity
• Instance Affinity
• Process Affinity
• Linux Process Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
136
1/24/2015 Midpoint Technology, Inc
CoreXL Manages 4 Affinity Types

VS1 VS2
2) Firewall
Instances

FWD syslogd
logging

SND
Secure
Network
Distributor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
137
1/24/2015 Midpoint Technology, Inc
HyperThread Review
Enabling Symmetric Multi-Threading (SMT) or Hyper Threading (HT) doubles the number of logical processors.

• Note the difference between HT, Dual Core and Dual Processor. Note where the caches are! Remember this
when you assign interfaces so you keep the cache hot
• Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting
pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel
processing.
• Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time.
• With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently.
So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time
slices.
• Performance improvement 30%???
Without Hyperthreading, Managed by Kernel
Thread 1 Thread 2 Thread 1 Thread 2

WithHyperthreading, Managed by HW

http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx

TIME

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
138
1/24/2015 Midpoint Technology, Inc
HT and Cache Sharing
Remember that the kernel instances share a state tables. So when allocating
instances keep similar data flows on the same cache so that portion of the connection
table is always in cache.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
139
1/24/2015 Midpoint Technology, Inc
R77.10 Uses SMT/HT

Implemented in R77+

Restrictions:
• Only enhances performance of IPS/AB/AV CPU intensive functions and
NOT I/O operations. Too many interrupts may actually slow it down.
• Supported only on R77+ GAIA
• Only on Check Point Appliances
• Has to be enabled in the BIOS
• Does not work with large number of HIDE NAT connections. Each CPU has
pre-allocated # of HIDE NAT slots. If one CPU uses all its HIDE NAT slots
then it can’t handle new HIDE NAT connections.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
140
1/24/2015 Midpoint Technology, Inc
Assigning Instances to CPUs

• NGF Firewall Affinity


• VSX Firewall Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
141
1/24/2015 Midpoint Technology, Inc
NGF Standalone Instance Affinity

In NGF there is only 1 firewall. With CoreXL, the kernel will replicate itself X times,
depending on how many firewall instances you setup. Each instance will parallel process
network traffic with the SAME shared rulebase and state tables.
Each instance has an ‘affinity’ for a specific processor.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
142
1/24/2015 Midpoint Technology, Inc
NGF CoreXL
In a NGF gateway, CoreXL
generates X copies of the kernel
into individual Linux kernel
threads
Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
143
1/24/2015 Midpoint Technology, Inc
NGF Process/Thread Mapping
Before we assign affinity, we should look at what types of processes/threads we
are dealing with. When multiple processes have the same PID, that means they
are threads sharing the VM of the parent (CPD and FWD below).

Here you can see the firewalls are individual KERNEL threads inside the kernel with a parent
of PID 1 – ‘init’. Kernel threads usually come in thread groups of size 1, unlike user space.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
144
1/24/2015 Midpoint Technology, Inc
NGF Process Spawn Tree
Here you can see how the firewall processes were spawned in what order

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
145
1/24/2015 Midpoint Technology, Inc
Assigning Instances to CPUs

• NGF Firewall Affinity


• VSX Firewall Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
147
1/24/2015 Midpoint Technology, Inc
VSX CoreXL Affinity
Where NGF has 1 firewall on a chassis, VSX has multiple firewalls running on a single physical chassis.
Each firewall is represented by a Virtual System (VS).

VSX VS’s are a different than NGF. Each VS has a 1 firewall instance that is executed by 1
corresponding Linux OS User mode process (not totally true, but not lying and still making the point).

VS VS VS VS

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
148
1/24/2015 Midpoint Technology, Inc
R75.40 VSX Process Architecture

VS instances are implemented with Linux threads (user mode but mapped onto
kernel threads so they can be scheduled by the kernel)

fw kernel VS

=
• Virtual Memory Space
• CPU Instructions instance instance
• Processor state
• Kernel threads
• Security attributes VS
instance

VS
instance

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
149
1/24/2015 Midpoint Technology, Inc
VS instances

The terminology gets a little VS


confusing because CP is not
explicit nor consistent in defining
what an instance is for VSX.

A firewall kernel instance which


runs on behalf of a VS, can also fw kernel VS
instance instance
be further subdivided into VS
instances. These VS instances can
VS
be assigned to individual instance
processors.
VS
instance

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
150
1/24/2015 Midpoint Technology, Inc
VSX VS0 CoreXL Affinity
Here you can see we are configured 3 VS instances within VS0 (very important!!).
I have 3 VS instances running. So you can see that VSi0-VSi2 are allowed to run on any
of the 1-3 CPUs. CPU 0 is reserved for eth0 traffic (hold on).

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
151
1/24/2015 Midpoint Technology, Inc
VS0 Processes and Threads
So VS0 has 4 VS instances (VSi) generated for it that are watched over by
the watcher daemon.

Processes Threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
152
1/24/2015 Midpoint Technology, Inc
VS0 CoreXL Config
You can only configure CoreXL VS0 from the command line….

Trying to configure more


VS instances for VS1 from
the command line and
look….

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
153
1/24/2015 Midpoint Technology, Inc
Change non-VS0 CoreXL
OK. We’ll listen to directions…

1= CoreXL OFF

2+ = CoreXL ON

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
154
1/24/2015 Midpoint Technology, Inc
Look Ma Babies!!!
• VS1 now has 2 instances.
• CPD has 1 main and 2 instances
• FWD has 4 daemons (not sure why)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
155
1/24/2015 Midpoint Technology, Inc
Linux view of VSX instances
VSX uses USER threads for all components

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
156
1/24/2015 Midpoint Technology, Inc
VSX Process Spawn Tree
Here you can see the parent-child relationship of how firewall instances are spawned.
Note: the process names change, like fw could be fwd – Linux command line issue.

pstree -p

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
157
1/24/2015 Midpoint Technology, Inc
Assigning Instances to CPUs

• NGF Firewall Affinity Assignment


• VSX Firewall Affinity Assignment
• Monitor Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
159
1/24/2015 Midpoint Technology, Inc
Default Affinity Assignment
The default affinity for firewall instances is to assign CPUs from high to low. There can never be more instances than
CPUs so that mapping is 1:1. The default affinity assignments for interfaces is to CPUs that are NOT running firewall
instances…if possible.

Note that the default assignment is probably adequate for 90% of the cases….unless you have a lot of busy network
interfaces, help processes, Linux processes that interfere with the firewall instances. For example, if eth0 ran 100% of
CPU0, then you might want to move the firewall instance.

From sk98348

# of Cores # of FW # of SNDs
Instances
1 1 0 (Corexl disabled)

2 2 2

4 3 1 Reserved for
SND
6-20 # cores -2 2

21-30 # cores -4 4
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
160
1/24/2015 Midpoint Technology, Inc
NGF: Assign Affinity
When you re-assign an instance to a CPU, you are telling the instance to only
use THAT CPU…when the CPU is free. So double-edge sword:
1. GOOD: Guarantee the cache will be always hot on that CPU
2. BAD: What if that CPU is busy with other assigned processes…Has to wait till
end of the other process timeslice to get CPU time. You could up its priority.
So make sure you choose a CPU that is NOT assigned to any other process if
possible.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
162
1/24/2015 Midpoint Technology, Inc
Assign ALL CPUs
You can set affinity to ALL CPUs by assigning all CPUs to the instance.

You can obviously also set the affinity to a subset of CPUs.

Note: that you forfeit hot caching

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
163
1/24/2015 Midpoint Technology, Inc
Affinity Set via Linux
Note that instead of fw ctl affinity command, you could just use the
regular Linux affinity command ‘taskset’. Does the same thing.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
164
1/24/2015 Midpoint Technology, Inc
Permanent Affinity
Custom firewall instance affinity are not permanent (but INTERFACE affinity is).
On reboot you have to re-assign affinity. You can make firewall instance affinity
permanent configuring it in $FWDIR/conf/fwaffinity.conf.

NOTE: auto vs. all

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
165
1/24/2015 Midpoint Technology, Inc
Stickiness
State table tuples are ‘sticky’ to a single core. Once you start a network
conversation through a specific core, the associated tuple will always use
the same CPU to process that network conversation.
So a backup process between backup server A and client B for example
could monopolize a single CPU.

<1.1.1.1, 1111, 2.2.2.2, 2222, TCP>

2.2.2.2 1.1.1.1
CORE 0

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
166
1/24/2015 Midpoint Technology, Inc
Assigning Instances to CPUs

• NGF Firewall Affinity Assignment


• VSX Firewall Affinity Assignment
• Monitor Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
167
1/24/2015 Midpoint Technology, Inc
VSX Default Affinity
The default config for VSX is firewall instances are assigned to all but CPU 0
(reserved for interfaces). Default varies depending on components, but it won’t be
“ALL”.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
168
1/24/2015 Midpoint Technology, Inc
VS Affinity Command Hierarchy
In VSX there is a hierarchy how VS affinity commands work. You can apply the instance
command to the ‘V’ entire VS, ‘P’, only the firewall processes or ‘I’ a specific VS instance.
The ‘I’ then inherits from the ‘P’ which inherits from the ‘VS’.

‘VS’ for Virtual System ‘P’ for Firewall Processes


‘I’ for instance
cpd

fwk

VS
fwd instance

vpnd

VS0
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
169
1/24/2015 Midpoint Technology, Inc
Impact of Setting Affinity

The SRC column shows at what level the affinity command for the process
was issued. ‘V’ means the command was issued to ALL the components
of a VS. ‘P’ means only to the firewall instances. ‘I’ means a single fw instance.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
170
1/24/2015 Midpoint Technology, Inc
VSX set affinity command
Here is the command for setting affinity for the 3 levels of processes.
Note that if you do not set a affinity for a level, the level will inherit
the affinity from the previous level.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
171
1/24/2015 Midpoint Technology, Inc
VS affinity config files
After you set affinity you can see the impact in these configuration files. Because
of these files, VSX affinity is permanent…unlike NGF!!!

Here are the affinity configurations that the VSs use to set their affinities. As
you set affinities at the different levels, these files will begin to appear.

This is how a VS instance knows what affinity to use. If there is no config file
at the I instance level, then it goes up to the P Process level config file, etc.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
172
1/24/2015 Midpoint Technology, Inc
Set VS affinity
Here we set the VS affinity. Note that it sets the affinity for ALL processes in VS0.
Firewalls, Firewall Helpers, Linux processes.
Note how the SRC column is “V” for the VS affinity configuration file

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
173
1/24/2015 Midpoint Technology, Inc
Set P Process Affinity
This will set the P Process affinity for JUST the fwk processes in a VS.
NOTE: the ‘P’ means to set affinity to what is found in the ‘P’ config file

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
174
1/24/2015 Midpoint Technology, Inc
Set P Process Affinity
This will set the P Process affinity is for JUST the fwk processes and VS instances.
NOTE: -fwkall will set the fwk VS instances for ALL the VS’s.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
175
1/24/2015 Midpoint Technology, Inc
Set ‘I’ VS Affinity
Here you can see we set he ‘I” VS Instance affinity

The “I” in the SRC column means the affinity config comes from the ‘I’
config file

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
176
1/24/2015 Midpoint Technology, Inc
Now let’s set everything to ALL
As with NGF, if you set affinity with all CPUs, it will be set to ‘ALL’. Note it only
sets the affinity for VS0, and not the other VSs

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from 177
1/24/2015 Midpoint Technology, Inc
VSX: CoreXL set per VS
CoreXL can be enabled/disabled per VS instance, just like SecureXL
VS0 use cpconfig, VS1+ use SmartDashboard to set CoreXL to 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
178
1/24/2015 Midpoint Technology, Inc
Reset to Defaults
REBOOT!!!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
179
1/24/2015 Midpoint Technology, Inc
Changing Cluster CoreXL config
At boot - Cluster membership is tested to ensure same # of CPUs in CoreXL.
When changing CoreXL cpconfig CPU configuration in a cluster:

1. Start on standby member B, bring it down


2. Change number of processors - cpconfig
3. Reboot
4. Member B Comes up in the READY state
5. Fail over Active member A to the Ready member B (Stateless)
6. Modify formerly Active member A
7. Reboot member A

SETTING AFFINITY (not CoreXL Firewall Instance count) must be done manually in both members
and does not impact cluster status

Member A - Active Member B - Ready

https://sc1.Check Point.com/documents/R76/CP_R76_ClusterXL_AdminGuide/7298.htm
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
180
1/24/2015 Midpoint Technology, Inc
Cluster XL Hints

1. Keep SecureXL and CoreXL configuration the same on both members


2. On upgrades, config will carry over but not necessarily on from scratch
migrations. ( e.g. process assignment)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
181
1/24/2015 Midpoint Technology, Inc
Summary: NGF to VSX CoreXL
NGF can delegate its firewall processing VSX has each VS represented by a Linux process with 1 or
into multiple fw instances. Each fw instance more internal VS instances mapped to Linux threads.
is represented by a Linux kernel thread The whole Linux process OR the VS instance (thread) can be
and can be assigned a core affinity. mapped to a processor

Linux
Process Linux Firewall Instance
/Firewall Process
Instance /Firewall
Instance
Firewall Instance Firewall Instance
Linux
Process
/Firewall Linux
Instance Process
/Firewall
Instance

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 182
Assigning Instances to CPUs

• NGF Firewall Affinity Assignment


• VSX Firewall Affinity Assignment
• Monitor Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
183
1/24/2015 Midpoint Technology, Inc
‘top’ Tricks for CoreXL

F SORT
j Processor
u Number of page faults

f Add
columns
j Processor
u Number of page faults
1 Show all processors
i,z Just show running processes
c Command vs process name

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
184
1/24/2015 Midpoint Technology, Inc
Important TOP Stats with ‘f’
command

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
185
1/24/2015 Midpoint Technology, Inc
Unix ‘top’
Show all processors and see what process is running on what processor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
186
1/24/2015 Midpoint Technology, Inc
Monitor Running Processes
With ‘f’ ->‘i’ and ‘j’ and ‘u’ you can monitor running processes that might be
faulting or jumping processors

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
187
1/24/2015 Midpoint Technology, Inc
CoreXL ‘ps’ arguments

-e All processes

-L All threads

-o Specific columns (below)

pid Pid of process

psr Processor id

cmd,comm,com Command/Process names in long-short formats. with


mand,args (args) arguments
lwp Thread ID

majflt Page faults

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
188
1/24/2015 Midpoint Technology, Inc
Unix ‘ps’ to monitor processor
‘watch –d –n1’ will execute a command every 1 second and highlight changes

‘ps’ can be used to monitor processor usage by a specific process

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
189
1/24/2015 Midpoint Technology, Inc
‘watch’ Interrupt table

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
191
1/24/2015 Midpoint Technology, Inc
CoreXL

• Interface Affinity
• Instance Affinity
• Process Affinity
• Linux Process Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
192
1/24/2015 Midpoint Technology, Inc
CoreXL Manages 4 Affinity Types

3) CP Process
VS1 VS2 4) Linux
Affinity Process

FWD syslogd
logging

SND
Secure
Network
Distributor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
193
1/24/2015 Midpoint Technology, Inc
List firewall helper daemons
Firewall helper daemons (cpd, fwd, vpnd) are running in the VS0-VSX VS’s.
Linux processes are running under the control of VS0.
Many times you will see they are able to run on ALL processors

Linux only processes

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
194
1/24/2015 Midpoint Technology, Inc
Set Process Affinity
Let’s say your fwd logging process is monopolizing the system with heavy
logging. You an assign a whole processor to it to handle the load.

NOTE: /opt/CPsuite-R75.40VS/fw1/conf/vsaffinity_exception.conf are a list of LINUX processes that are not impacted by
the affinity command. You have to edit this list to modify LINUX process affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
196
1/24/2015 Midpoint Technology, Inc
Process to CPU
Busy processes such as VPND or FWD (logging) can be dedicated
to a single CPU because they might dominate a core or swap cores
frequently thus losing their cache freshness.

Note in VSX it is specific to your VS context

FWD
Logging/HA

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
197
1/24/2015 Midpoint Technology, Inc
CoreXL Licensing

• Appliances include all the cores


• Open Servers
• You can select how many cores to buy
• CPSG-PCC
• CC = # of cores
• CPSG-PCCBB
• BB = # of blades
• CPSG-P1207
• Security Gateway
• 12 core
• 7 blades
• NOTE: You only have to license the cores that FWK run on. You can
buy a 2 core license and have a 12 core gateway. ..10 of the cores
will run Linux, SND, CP helper programs. 2 cores run FWK
• Example: Ala carte 2 core to 4 core upgrade $7500 list

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 198
Balancing Check Point Systems

• Overview
• Linux Review
• SecureXL
• CoreXL
• Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
199
1/24/2015 Midpoint Technology, Inc
Tuning Tips

• Traffic to Rules to SecureXL to CoreXL tuning


• VSX: Keep traffic flows on same system
• VLAN resources
• CPU or Memory
• IPS Integration???
• Suggested CoreXL Configurations
• Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
200
1/24/2015 Midpoint Technology, Inc
Classic Target

2 interface Internet firewall, busy at noon with HTTP traffic interrupting


business traffic

Symptom Verify

Highly utilized system ‘top’, ps –o psr,command

High network traffic on 2 interfaces /proc/interfaces

High HTTP traffic SmartLog

Ifconfig is dropping packets, retransmits Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
201
1/24/2015 Midpoint Technology, Inc
Determine Your Traffic Patterns

eth0 to eth3

Use SmartLog to determine your major traffic flows


between interfaces.
Also look for busy tuples, remember they are sticky to
processor!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
202
1/24/2015 Midpoint Technology, Inc
SecureXL Restrictions

The following traffic is not throughput (state tables), nor connection-rate (accept
templates)accelerated by SecureXL:
Traffic types other than TCP, UDP, PIM, GRE, ESP
First packets of any new TCP session, unless a "template" exists
First packet in a UDP session
Traffic matching certain Firewall rules:
rules with a service that uses a resource
rules for dropping or rejecting traffic
rules where the source or destination is the gateway itself
rules with a Security Server
rules with user authentication
rules with session authentication

The following traffic is not connection-rate (accept templates) accelerated by SecureXL and will stop
building templates in the rulebase if they are found:
Non-TCP/UDP connections such as PIM, GRE, ESP ---- ICMP
Protocols that are not connection intensive such as SMTP, FTP, RPC, NFS, NNTP, NTP
Complex connections such as IPSec VPN, FTP, H.323, etc.
Traffic in environment using NAT (for security, NAT addresses can change and can be shared)
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
203
1/24/2015 Midpoint Technology, Inc
Check SecureXL and Rulebase
ICMP prevents SecureXL Accept Connection templates from accelerating HTTP 1.0/type
connections. Need to move to end of rulebase.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
204
1/24/2015 Midpoint Technology, Inc
HTTPS and CoreXL tuning
Use /proc/interrupts and ifconfig to see if any of your interfaces were
struggling to keep up. If the interfaces are constantly rebalancing or
dropping packets, then you might have a problem.

All interfaces
on eth0!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
205
1/24/2015 Midpoint Technology, Inc
Processor Utilization
Look to see if CPU 0 is busy.

Busy FWD

Busy core 0

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
206
1/24/2015 Midpoint Technology, Inc
Balance CoreXL
So investigation shows Core 0 is busy. This means you should dedicate a ‘free’
processor to eth0 because eth0 is doing a lot of work with regular processes AND
interface handling

Give eth0 and eth3


their own core

Then give the offending process its own processor.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
207
1/24/2015 Midpoint Technology, Inc
Drop Rule
If SmartTracker or SmartEvent shows a Denial of Service or heavy traffic from a
malicious source, consider using drop templates or Suspicous Activity Monitor
instead of the rulebase to drop
the traffic. The drop templates are enforced by SecureXL and done before you
have to go through the rulebase.
https://supportcenter.Check Point.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67861

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
208
1/24/2015 Midpoint Technology, Inc
Tuning Tips

• Traffic to Rules to SecureXL to CoreXL tuning


• VSX: Keep traffic flows on same system
• VLAN resources
• CPU or Memory?
• Suggested CoreXL Configurations
• Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
209
1/24/2015 Midpoint Technology, Inc
VSX Archtecture

With VSX which is the most efficient way to route


packets between two networks and why?

Router
Router

VSX VSX VSX VSX


Chassis Chassis VRouter Chassis Chassis
VRouter VRouter

11 1 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
1/24/2015 Midpoint Technology, Inc 210
VSX Architecture Design

Balance VSLS VSs based on traffic flow w/ SecureXL caching


If you have lots of traffic between two VSs, then keep them on the same
gateway because SecureXL /CoreXL will cache states and accelerate locally

VSX VSX
Traffic states cached Chassis Chassis
locally and SecureXL will
not send it through user
mode kernel.
CoreXL will cache states in
processor CPU dedicated
to those interfaces

SND SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Tuning Tips

• Traffic to Rules to SecureXL to CoreXL tuning


• VSX: Keep traffic flows on same system
• VLAN resources
• CPU or Memory?
• Suggested CoreXL Configurations
• Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
212
1/24/2015 Midpoint Technology, Inc
Bonded(VLAN) Interfaces
When you have a bonded VLAN that trunks multiple networks over a single
logical interface, (assuming it will be heavily utilized) considering assigning
a dedicated CPU to each interface to handle the load and keep the caches
‘fresh’.

VLAN
VLAN

VLAN

VLAN

Core 0 Core 1
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com
Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
Tuning Tips

• Traffic to Rules to SecureXL to CoreXL tuning


• VSX: Keep traffic flows on same system
• VLAN resources
• CPU or Memory
• Suggested CoreXL Configurations
• Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
214
1/24/2015 Midpoint Technology, Inc
CPU or Memory

How do you know if you need more CPU or more memory or


both?

? ?

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
215
1/24/2015 Midpoint Technology, Inc
‘top’ is WRONG
Linux 2.6.18 ‘top’ is just WRONG. It does not compute SWAP space correctly
because it does not take into account (e.g.) that the processes is sharing 2GB
of space from a shared library along with 10 other processes. So each of the 10
processes thinks it alone is using the 2GB of shared space.
Below you can see that nautilus VIRT is 433MB, and SWAP is 427Mb, but so are
most the other processes. It just doesn’t add up to the ‘used’ SWAP.
So its hard to tell what processes are using what memory and what is using SWAP space.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
216
1/24/2015 Midpoint Technology, Inc
#1 Mo Memory
Because you can’t trust ‘top’ too much, a better measure is to look at the major
page faults for running processes. If they are faulting a lot, you have a problemo.

Using ‘top’ watch the running processes by typing ‘i’. This will only list the running
processes. Then use ‘f’ and ‘u’ for listing page faults.

Below is a real live MLM that is faulting heavily on one of the logging daemons. You can
have a fast CPU and add more CPUs, but fwd will spend most of its time swapping pages.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
217
1/24/2015 Midpoint Technology, Inc
#2 More CPU

If you have a lot of active processes that are NOT faulting, then you have 3
options:

1. More CPUs
2. Faster CPUs
3. Set affinity on the most busiest processes so they are using their hot caches
and not spending time flushing caches
4. Set the affinity of the busiest processes to -15 so that they get a bigger chunk
of the timeslice

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
218
1/24/2015 Midpoint Technology, Inc
#3 Memory Leak
1. If ‘Swap used” is > 0 and growing, you may have a memory leak.
I
2. If “VIRT” is growing over time on a process, you may have a memory leak.

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
219
1/24/2015 Midpoint Technology, Inc
Tuning Tips

• Traffic to Rules to SecureXL to CoreXL tuning


• VSX: Keep traffic flows on same system
• VLAN resources
• CPU or Memory
• Suggested CoreXL Configurations
• Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
220
1/24/2015 Midpoint Technology, Inc
Classic Target - NGF
2 interface Internet firewall, busy at noon with HTTP traffic interrupting
business trafficAssumes that SecureXL is working on ALL rules and SND is doing
most the work and not the FWK. NO IPS/AV/ThreatPrevention,etc.

Goal is to allocate evenly. Obviously this is only a guess, but you’d have
to evaluate your system with the following commands to have more accurate
measurement

Symptom Verify

Highly utilized system ‘top’, ps –o psr,command

High network traffic on 2 interfaces /proc/interfaces

High HTTP traffic SmartLog

Ifconfig is dropping packets, retransmits Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
221
1/24/2015 Midpoint Technology, Inc
Balancing Goals
This is the take-away slide, the value behind the whole course.

Allocate CPUs in the following priority order:

1. Share cache for common data


2. Allocate CPUs to busy Internal interfaces
3. Allocate CPUs to slower less busy External interfaces
4. Allocate CPUs to FW instances
5. Remaining threads are usually idle so distribute evenly
and let kernel find idle processor

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
222
1/24/2015 Midpoint Technology, Inc
Reminder: Share Cache!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
223
1/24/2015 Midpoint Technology, Inc
Suggested CoreXL Distributions-NGF
Type CPU/Item 1 2 4 8 (2 core) 12 (2 core)

FW cpd 0 All All 3 3


Processes
fwd 0 All All 3 4

Linux Linux All All All All All


processes

Interfaces External 0 0 0 0 (cache 0) 0 (cache 0)


(SND) Interface

Internal 0 1 1 1 (cache 0) 1 (cache 0)


Interface
Sync 0 All All 2 2

Fw kernel fw_0 0 0 2 4(cache 1) 6(cache 1)


instances
fw_1 1 3 5(cache 1) 7(cache 1)

fw_2 6(cache 1) 8(cache 1)

fw_3 7(cache 1) 9(cache 1)

fw_4 10(cache 1)

fw_5 11 (cache 1)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
224
1/24/2015 Midpoint Technology, Inc
Classic Target - VSX
VSX with multiple business silos to Internet. Assumes that SecureXL is working
on ALL rules and SND is doing most the work and not the FWK. NO IPS/AV,etc
NOTE: fw0 is not handling any traffic aside from sync. Only mgt traffic

Goal is to allocate evenly. Obviously this is only a guess, but you’d have
to evaluate your system with the following commands to have more accurate
measurement

Symptom Verify

Highly utilized system ‘top’, ps –o psr,command,


vsx resctrl, vsx memstat
High network traffic on 2 interfaces /proc/interfaces

High HTTP traffic SmartLog

Ifconfig is dropping packets, retransmits Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
225
1/24/2015 Midpoint Technology, Inc
Suggested CoreXL Distributions-VSX
Type CPU/Item 1 2 4 8 (dual core) 12(dual
core)
Interfaces Ext VS0 0 All 0 4 6
(SND)
Ext VS1 0 0 1 1(cache0) 0 (cache0)

Ext VS2 0 1 2 1(cache0) 1 (cache0)

Ext VS3 0 1 3 3(cache0) 2 (cache0)

Int VS0 0 All 0 4 6

Int VS1 0 0 1 0(cache0) 3 (cache0)

Int VS2 0 1 2 1(cache0) 4 (cache0)

Int VS3 0 1 3 2(cache0) 5 (cache0)

Sync 0 All 0 4 7

Fw kernel fw_0 0 All 0 4 8


instances
fw_1 (VS1) 0 All 1 5 9

fw_2 (VS2) 0 All 2 6 10

fw_3 (VS3) 0 All 3 7 11

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
226
1/24/2015 Midpoint Technology, Inc
Suggested CoreXL Distributions-VSX

Type CPU/Item 1 2 4 8 (dual core) 12(dual


core)
cpd(VS1) All All All but 0-3 6,7,8

cpd(VS2) All All All but 0-3 6,7,8

cpd(VS3) All All All but 0-3 6,7,8

fwd(VS0) All All All but 0-3 6,7,8

fwd(VS1) All All All but 0-3 6,7,8

fwd(VS2) All All All but 0-3 6,7,8

fwd(VS3) All All All but 0-3 6,7,8

Linux Linux All All All but 0-3 All


processes

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
227
1/24/2015 Midpoint Technology, Inc
Tuning Tips

• Traffic to Rules to SecureXL to CoreXL tuning


• VSX: Keep traffic flows on same system
• VLAN resources
• Suggested CoreXL Configurations
• Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
228
1/24/2015 Midpoint Technology, Inc
SecureXL Cheat Sheet

sim affinity –l List interface affinities


NGF: fw ctl affinity –l –v -a List all affinities including
interfaces
VSX: fw ctl affinity –l -x List all affinities including
interfaces
sim affinity -s Set interface affinities
fwaccel stat Review SecureXL status
fwaccel stats Review SecureXL stats
fwaccel conns Review SecureXL state table
fwaccel templates Review SecureXL accept
connection templates

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
229
1/24/2015 Midpoint Technology, Inc
CoreXL Commands

List interface affinities


NGF: fw ctl affinity –l –v –a List all affinities including
fw ctl multik stat interfaces
VSX: fw ctl affinity –l –x List all affinities including
fw ctl affinity –l -x –flags tkn interfaces.
fw ctl multik stat -flags tkn lists internal threads
cat /proc/interrupts List IRQ table and interrupts
handled by CPU
watch –d –n1 “ps –ef | fgrep Watch how a process changes CPU
syslogd”
top -------- F->j
fw ctl affinity –s –d –fwkall 3 Set all process affinities to ALL
(auto)

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
230
1/24/2015 Midpoint Technology, Inc
Thanks!!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com


Proprietary and Confidential: No part of this document may be reproduced without permission from
231
1/24/2015 Midpoint Technology, Inc