The Magic of

IP Flow
Valens Riyadi
info@mikrotik.co.id
Citraweb Nusa Infomedia
on Mikrotik User Meeting, Krakow
January 25 – 26, 2007
 Introduction
| Name: Valens Riyadi
| Country: Indonesia
z Graduated as Architect 1998
z 1998 ….. Web developer
z 2001 ….. Make a WISP
z 2002 ….. Mikrotik Reseller
z Photographer
• Administrator of www.fotografer.net
z Head of Security Dept, Indonesian ISP Association
z Volunteer for Airputih Foundation, IT Emergency Task Force
z Steering Committee for ID-SIRTII
Indonesia Security Incident Response Team on Information Infrastructure
z Mikrotik Certified Consultant

00-2 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 My Company
| Citraweb Nusa Infomedia
z Web Developer (since 2000)
z Small ISP (since 2001)
z Mikrotik Reseller (since 2002)

| Located at : Yogyakarta Indonesia

| Using RouterOS since 2.3.15

00-3 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Yogyakarta City
| 3,4 million of population
z Tourism City
z Student City
• Almost 50% of population are students from other cities.
z Finally ……. Cyber café City

00-4 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Network Topology
INTERNET

INDONESIA-IX
GATEWAY
ROUTER

YOGYA-IX
PROXIES E1 ROUTER

BANDWIDTH
MANAGEMENT

SERVERS TO
CUSTOMER

DISTRIBUTION
ROUTER

INTERNAL NAT
ROUTER
00-5 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
 Wireless Instalation

00-6 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Wireless Network Topology
BTS2 BTS3 BTS4

BTS5

BTS1

BTS6

NOC-1 NOC-2
Ethernet Cable

Main Wireless Link

Distribution Backup Wireless Link
Router
00-7 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
 Fail Over Scenario (1)
BTS2 BTS3 BTS4

BTS5

BTS1

DOWN
BTS6

DOWN
NOC-1 NOC-2
Ethernet Cable

Main Wireless Link

Distribution Backup Wireless Link
Router
00-8 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
 Fail Over Scenario (2)
BTS2 BTS3 BTS4

BTS5

BTS1

BTS6
DOWN
X
NOC-1 NOC-2
Ethernet Cable

Main Wireless Link

Distribution Backup Wireless Link
Router
00-9 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
The Basic of
IP Flow
 IP Flow (simple diagram)
INPUT PRE POST OUTPUT
FORWARD
INTERFACE ROUTING ROUTING INTERFACE

LOCAL
INPUT OUTPUT
PROCESS

PREROUTING INPUT FORWARD OUTPUT POSTROUTING

Hotspot Input Mangle Mangle Conn-Tracking Mangle
Conn-Tracking Filter Filter Mangle Global-Out Queue
Mangle Acounting Filter Global-Total Queue
Dst-NAT Source-NAT
Global-In Queue Hotspot Output
Global-Total Queue

00-11 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 OUTPUT POSTROUTING
Conn-Tracking Mangle

IP Flow Mangle
Filter
Global-Out Queue
Global-Total Queue
Source-NAT
+ FORWARD Hotspot Output
BRIDGE Bridge BRIDGE Mangle
DST-NAT Decision FORWARD Filter
PRE Acounting
- ROUTING
-
Broute?
+ BRIDGE
INPUT
+ FORWARD
INPUT is - Routing
Bridged? Decision

Routing OUTPUT is + Bridge

INPUT INPUT Decision Bridged? Decision
INTERFACE
-
IPSEC + IPsec
OUTPUT
BRIDGE
DECRYPTION Policy OUTPUT
- POST
ROUTING
BRIDGE
LOCAL LOCAL
PREROUTING SRC-NAT
PROCESS-IN PROCESS-OUT
Hotspot Input
Conn-Tracking
Mangle IPSEC + IPsec
Dst-NAT INPUT
ENCRYPTION Policy
Global-In Queue Mangle
Global-Total Queue Filter -
INTERFACE OUTPUT
00-12 Mikrotik Indonesia http://www.mikrotik.co.id QUEUE 1/18/2007
INTERFACE
 From – To Traffic?
| For each data packet, you have to know:
z Source of packet
• From outside
• From local Process
z Destination of packet
• To Local Process
• To outside

00-13 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 OUTPUT POSTROUTING
Routed Traffic Conn-Tracking
Mangle
Mangle
Global-Out Queue

To Router Filter

FORWARD
Global-Total Queue
Source-NAT
Hotspot Output
BRIDGE Bridge + BRIDGE Mangle
DST-NAT Decision FORWARD Filter
PRE Acounting
- ROUTING
-
Broute?
+ BRIDGE
INPUT
+ FORWARD
INPUT is - Routing
Bridged? Decision

Routing OUTPUT is + Bridge

INPUT INPUT Decision Bridged? Decision
INTERFACE
-
IPSEC + IPsec
OUTPUT
BRIDGE
DECRYPTION Policy OUTPUT
- POST
ROUTING
BRIDGE
LOCAL LOCAL
PREROUTING SRC-NAT
PROCESS-IN PROCESS-OUT
Hotspot Input
Conn-Tracking
Mangle IPSEC + IPsec
Dst-NAT INPUT
ENCRYPTION Policy
Global-In Queue Mangle
Global-Total Queue Filter -
INTERFACE OUTPUT
00-14 Mikrotik Indonesia http://www.mikrotik.co.id QUEUE 1/18/2007
INTERFACE
 OUTPUT POSTROUTING
Routed Traffic Conn-Tracking
Mangle
Mangle
Global-Out Queue

From Router Filter Global-Total Queue

Source-NAT
+ FORWARD Hotspot Output
BRIDGE Bridge BRIDGE Mangle
DST-NAT Decision FORWARD Filter
PRE Acounting
- ROUTING
-
Broute?
+ BRIDGE
INPUT
+ FORWARD
INPUT is - Routing
Bridged? Decision

Routing OUTPUT is + Bridge

INPUT INPUT Decision Bridged? Decision
INTERFACE
-
IPSEC + IPsec
OUTPUT
BRIDGE
DECRYPTION Policy OUTPUT
- POST
ROUTING
BRIDGE
LOCAL LOCAL
PREROUTING SRC-NAT
PROCESS-IN PROCESS-OUT
Hotspot Input
Conn-Tracking
Mangle IPSEC + IPsec
Dst-NAT INPUT
ENCRYPTION Policy
Global-In Queue Mangle
Global-Total Queue Filter -
INTERFACE OUTPUT
00-15 Mikrotik Indonesia http://www.mikrotik.co.id QUEUE 1/18/2007
INTERFACE
 OUTPUT POSTROUTING
Routed Traffic Conn-Tracking
Mangle
Mangle
Global-Out Queue

Through Router Filter

FORWARD
Global-Total Queue
Source-NAT
Hotspot Output
BRIDGE Bridge + BRIDGE Mangle
DST-NAT Decision FORWARD Filter
PRE Acounting
- ROUTING
-
Broute?
+ BRIDGE
INPUT
+ FORWARD
INPUT is - Routing
Bridged? Decision

Routing OUTPUT is + Bridge

INPUT INPUT Decision Bridged? Decision
INTERFACE
-
IPSEC + IPsec
OUTPUT
BRIDGE
DECRYPTION Policy OUTPUT
- POST
ROUTING
BRIDGE
LOCAL LOCAL
PREROUTING SRC-NAT
PROCESS-IN PROCESS-OUT
Hotspot Input
Conn-Tracking
Mangle IPSEC + IPsec
Dst-NAT INPUT
ENCRYPTION Policy
Global-In Queue Mangle
Global-Total Queue Filter -
INTERFACE OUTPUT
00-16 Mikrotik Indonesia http://www.mikrotik.co.id QUEUE 1/18/2007
INTERFACE
 OUTPUT POSTROUTING
Bridge Traffic Conn-Tracking
Mangle
Mangle
Global-Out Queue

Through Router Filter Global-Total Queue

Source-NAT
+ FORWARD Hotspot Output
BRIDGE Bridge BRIDGE Mangle
DST-NAT Decision FORWARD Filter
PRE Acounting
- ROUTING
-
Broute?
+ BRIDGE
INPUT
+ FORWARD
INPUT is - Routing
Bridged? Decision

Routing OUTPUT is + Bridge

INPUT INPUT Decision Bridged? Decision
INTERFACE
-
IPSEC + IPsec
OUTPUT
BRIDGE
DECRYPTION Policy OUTPUT
- POST
ROUTING
BRIDGE
LOCAL LOCAL
PREROUTING SRC-NAT
PROCESS-IN PROCESS-OUT
Hotspot Input
Conn-Tracking
Mangle IPSEC + IPsec
Dst-NAT INPUT
ENCRYPTION Policy
Global-In Queue Mangle
Global-Total Queue Filter -
INTERFACE OUTPUT
00-17 Mikrotik Indonesia http://www.mikrotik.co.id QUEUE 1/18/2007
INTERFACE
 Chain Position
From To Mangle Firewall Queue
Outside Router / Prerouting Global-in
Local Input Input Global-Total
process
Router/ Outside Output Output Global-Out
Local Postrouting Global-Total
process Interface
Outside Outside Prerouting Global-in
Forward Forward Global-out
Postrouting Global-total
Interface
00-18 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
 Simple Queue
| Simple Queue is located at Global-In and
Global-Out…. and also at Global Total

00-19 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Mangle & Simple Queue
| Mangle
z chain=forward in-interface=LAN
src-address=192.168.0.4 action=mark-packet
new-packet-mark=client passthrough=no
z chain=forward out-interface=LAN
dst-address=192.168.0.4 action=mark-packet
new-packet-mark=client passthrough=no
| Simple Queue
z name="queue1" interface=all parent=none
packet-marks=client direction=both
max-limit=512000/512000

00-20 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 IP Flow (simple diagram)
INPUT PRE POST OUTPUT
FORWARD
INTERFACE ROUTING ROUTING INTERFACE

QUEUE DOWNLOAD
QUEUE UPLOAD

GLOBAL-OUT
DOWNLOAD
LOCAL
GLOBAL-IN
INPUT OUTPUT

MANGLE

MANGLE
UPLOAD
PROCESS

PREROUTING INPUT FORWARD OUTPUT POSTROUTING

Hotspot Input Mangle Mangle Conn-Tracking Mangle
Conn-Tracking Filter Filter Mangle Global-Out Queue
Mangle Acounting Filter Global-Total Queue
Dst-NAT Source-NAT
Global-In Queue Hotspot Output
Global-Total Queue

00-21 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Mangle & Simple Queue
| This sample :
z will work for download limiting
z will not work for upload limiting
• because mangle will be done after simple queue
process
• mangle : chain=forward
• simple queue Æ global-in (prerouting)
z mangle should be in prerouting (for upload)
and postrouting (for download)

00-22 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 IP Flow (simple diagram)
INPUT PRE POST OUTPUT
FORWARD
INTERFACE ROUTING ROUTING INTERFACE

QUEUE DOWNLOAD
QUEUE UPLOAD

GLOBAL-OUT
DOWNLOAD
GLOBAL-IN LOCAL
INPUT OUTPUT
MANGLE

MANGLE
UPLOAD

PROCESS

PREROUTING INPUT FORWARD OUTPUT POSTROUTING

Hotspot Input Mangle Mangle Conn-Tracking Mangle
Conn-Tracking Filter Filter Mangle Global-Out Queue
Mangle Acounting Filter Global-Total Queue
Dst-NAT Source-NAT
Global-In Queue Hotspot Output
Global-Total Queue

00-23 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

Test Case (1)

Transparant
Bandwidth Management
 Queue with Bridge
BRIDGE

BRIDGE

Traffic Client - Internet INTERNET

QUEUE TREE

00-25 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Queue with Bridge
BRIDGE

BRIDGE

Upstream

INTERNET
Downstream

QUEUE TREE

00-26 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Interface Setup
[admin@MikroTik] > in pr
Flags: X - disabled, D - dynamic, R - running
# Name Type RX-RATE TX-RATE MTU
0 R LAN ether 0 0 1500
1 R WAN ether 0 0 1500
2 R bridge1 bridge 0 0 1500
[admin@MikroTik] interface bridge port> pr
Flags: X - disabled, I - inactive, D - dynamic
# INTERFACE BRIDGE PRIORITY PATH-COST

0 WAN bridge1 128 10

1 LAN bridge1 128 10

00-27 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Mangle Setup
[admin@MikroTik] > ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting in-interface=LAN
src-address=192.168.0.0/24 action=mark-packet
new-packet-mark=data-up passthrough=no

1 chain=postrouting out-interface=LAN
dst-address=192.168.0.0/24 action=mark-packet
new-packet-mark=data-down passthrough=no

00-28 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Queue Tree Setup
[admin@MikroTik] > queue tree print
Flags: X - disabled, I - invalid
0 name="queue-up" parent=WAN
packet-mark=data-up limit-at=512000
queue=default priority=8 max-limit=512000
burst-limit=0 burst-threshold=0 burst-time=0s
1 name="queue-down" parent=LAN
packet-mark=data-down limit-at=1024000
queue=default priority=8 max-limit=1024000
burst-limit=0 burst-threshold=0 burst-time=0s

00-29 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

Test Case (2)

Queue with
Src-NAT and Internal Proxy
 Queue with
SRC-NAT & Internal Proxy
ROUTER

SRC-NAT

Traffic Client - Internet INTERNET

WEB-PROXY
LOCAL
PROCESS

00-31 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Queue with
SRC-NAT & Internal Proxy
ROUTER

Direct Upstream 1
SRC-NAT
2
Direct Downstream

5 INTERNET
3
Upstream to proxy
WEB-PROXY
LOCAL
PROCESS
Downstream from proxy
4 6

00-32 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Web-Proxy Setup
> ip web-proxy pr enabled: yes
src-address: 0.0.0.0
port: 3128
hostname: "proxy"
transparent-proxy: yes
parent-proxy: 0.0.0.0:0
cache-administrator: "webmaster"
max-object-size: 4096KiB
cache-drive: system
max-cache-size: none
max-ram-cache-size: unlimited
status: running
reserved-for-cache: 0KiB
reserved-for-ram-cache: 154624KiB
00-33 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
 Firewall Setup
| [admin@instaler] ip firewall nat> pr
Flags: X - disabled, I - invalid, D - dynamic
0 chain=srcnat out-interface=public
src-address=192.168.1.0/24
action=masquerade
1 chain=dstnat in-interface=lan
src-address=192.168.1.0/24 protocol=tcp
dst-port=80 action=redirect to-ports=3128

00-34 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Mangle Setup
0 ;;; UP TRAFFIC / Traffic #1 and #3
chain=prerouting in-interface=lan
src-address=192.168.1.0/24 action=mark-packet
new-packet-mark=test-up passthrough=no
1 ;;; CONN-MARK
chain=forward src-address=192.168.1.0/24 action=mark-
connection new-connection-mark=test-conn passthrough=yes
2 ;;; DOWN-DIRECT CONNECTION / Traffic #2
chain=forward in-interface=public
connection-mark=test-conn action=mark-packet
new-packet-mark=test-down passthrough=no
3 ;;; DOWN-VIA PROXY / Traffic #4
chain=output out-interface=lan dst-address=192.168.1.0/24
action=mark-packet new-packet-mark=test-down
passthrough=no
00-35 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007
 Queue Setup
0 ;;; For traffic #2 and #4 (download)
name="downstream" parent=lan
packet-mark=test-down limit-at=1024000
queue=default priority=8 max-limit=1024000
burst-limit=0 burst-threshold=0 burst-time=0s
1 ;;; For traffic #1 and #3 (upload)
name="upstream" parent=global-in
packet-mark=test-up limit-at=256000
queue=default priority=8 max-limit=256000
burst-limit=0 burst-threshold=0 burst-time=0s

00-36 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Traffic #5 & #6
| We can not manage traffic #5 and #6 based
on client IP Address, because after the
traffic hits the proxy, it will change the
source IP Address, and the traffic will be a
new one:
z Source : Web Proxy (local process)
z Destination : Web Server on Internet

00-37 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007

 Thank You
| Valens Riyadi
| info@mikrotik.co.id

00-38 Mikrotik Indonesia http://www.mikrotik.co.id 1/18/2007