Nexus Technology Labs Classical Ethernet Switching

Port Channels
Last updated: April 11, 2013

Task
Configure port channels between N7K1, N7K2, N5K1, and N5K2 as follows: The links between N5K1 and N7K1 should be grouped together in Port-Channel 1. Do not use a negotiation protocol for this channel. The links between N7K1 and N7K2 should be grouped into Port-Channels 2 and 3. Both switches should initiate LACP negotiation for both of these channels and use LACP fast timers. The links between N5K2 and N7K2 should grouped together into Port-Channel 4. N7K2 should initiate LACP negotiation and N5K2 should respond. Disable all other interconnections between the switches. Configure all of these port channels as 802.1Q trunk links. Configure N7K2 with an LACP priority of 16384 so that it is the preferred device for managing negotiation of its port channels. Configure all switches to use source and destination TCP/UDP ports for load balancing flows across the port channel members. Configure N5K1's link to Server 1 and N5K2's link to Server 2 in VLAN 10. Server 1 should use the IP address 10.0.0.1/24 and Server 2 should use the IP address 10.0.0.2/24. When complete, Server 1 and Server 2 should have IP reachability to each other. Use the iPerf application to verify that traffic flows between Server 1 and Server 2 are being distributed among the member links of the port channels.

Configuration
N 5 K 1 : v l a n1 0 ! p o r t c h a n n e ll o a d b a l a n c ee t h e r n e ts o u r c e d e s t p o r t ! i n t e r f a c ep o r t c h a n n e l 1 s w i t c h p o r tm o d et r u n k s p e e d1 0 0 0 0

FEEDBACK

! i n t e r f a c eE t h e r n e t 1 / 1 s w i t c h p o r ta c c e s sv l a n1 0 s p e e d1 0 0 0 ! i n t e r f a c eE t h e r n e t 1 / 2-6 s h u t d o w n ! i n t e r f a c eE t h e r n e t 1 / 6 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p1 ! i n t e r f a c eE t h e r n e t 1 / 7 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p1 ! i n t e r f a c eE t h e r n e t 1 / 8-9 s h u t d o w n N 5 K 2 : f e a t u r el a c p ! v l a n1 0 ! p o r t c h a n n e ll o a d b a l a n c ee t h e r n e ts o u r c e d e s t p o r t ! i n t e r f a c ep o r t c h a n n e l 4 s w i t c h p o r tm o d et r u n k s p e e d1 0 0 0 0 ! i n t e r f a c eE t h e r n e t 1 / 1 s h u t d o w n ! i n t e r f a c eE t h e r n e t 1 / 2 s w i t c h p o r ta c c e s sv l a n1 0 s p e e d1 0 0 0 ! i n t e r f a c eE t h e r n e t 1 / 3-5 s h u t d o w n ! i n t e r f a c eE t h e r n e t 1 / 6 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p4m o d ep a s s i v e ! i n t e r f a c eE t h e r n e t 1 / 7 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p4m o d ep a s s i v e ! i n t e r f a c eE t h e r n e t 1 / 8-9 s h u t d o w n

N 7 K 1 : f e a t u r el a c p ! v l a n1 0 ! p o r t c h a n n e ll o a d b a l a n c es r c d s tl 4 p o r t ! i n t e r f a c ep o r t c h a n n e l 1 s w i t c h p o r t s w i t c h p o r tm o d et r u n k ! i n t e r f a c ep o r t c h a n n e l 2 s w i t c h p o r t s w i t c h p o r tm o d et r u n k ! i n t e r f a c ep o r t c h a n n e l 3 s w i t c h p o r t s w i t c h p o r tm o d et r u n k ! i n t e r f a c eE t h e r n e t 1 / 1 l a c pr a t ef a s t s w i t c h p o r t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p2m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 1 / 2 l a c pr a t ef a s t s w i t c h p o r t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p2m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 1 l a c pr a t ef a s t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p3m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 2 l a c pr a t ef a s t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p3m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 3 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p1 n os h u t d o w n

! i n t e r f a c eE t h e r n e t 2 / 4 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p1 n os h u t d o w n N 7 K 2 : f e a t u r el a c p ! v l a n1 0 ! l a c ps y s t e m p r i o r i t y1 6 3 8 4 p o r t c h a n n e ll o a d b a l a n c es r c d s tl 4 p o r t ! i n t e r f a c ep o r t c h a n n e l 2 s w i t c h p o r t s w i t c h p o r tm o d et r u n k ! i n t e r f a c ep o r t c h a n n e l 3 s w i t c h p o r t s w i t c h p o r tm o d et r u n k ! i n t e r f a c ep o r t c h a n n e l 4 s w i t c h p o r t s w i t c h p o r tm o d et r u n k ! i n t e r f a c eE t h e r n e t 1 / 1 l a c pr a t ef a s t s w i t c h p o r t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p2m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 1 / 2 l a c pr a t ef a s t s w i t c h p o r t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p2m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 1 l a c pr a t ef a s t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p3m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 2 l a c pr a t ef a s t s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p3m o d ea c t i v e

n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 3 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p4m o d ea c t i v e n os h u t d o w n ! i n t e r f a c eE t h e r n e t 2 / 4 s w i t c h p o r tm o d et r u n k c h a n n e l g r o u p4m o d ea c t i v e n os h u t d o w n

Verification
Port channels in NX-OS, just like in Catalyst IOS and other platforms, require that the member interfaces first have compatible parameters for the channel to form. In NX-OS, these parameters can be verified with the command s h o wp o r t c h a n n e lc o m p a t i b i l i t y p a r a m e t e r s. Some of these parameters can be seen below:

N 7 K 1 1 #s h o wp o r t c h a n n e lc o m p a t i b i l i t y p a r a m e t e r s|i n c l u d e\ * *p o r tm o d e *s p e e d *M T U *M E D I U M *S p a nm o d e < s n i p >

In this topology, the links between the Nexus 7000 switches include both M series and F series modules. Because these modules have different port level capabilities, they are not compatible to channel together. This is why the above solution has one port channel for the M1 ports and another for the F1 ports. The NX-OS parser will detect this and return an error message if you attempt to channel together incompatible port types, as shown below:

N 7 K 1 1 #c o n f i gt E n t e rc o n f i g u r a t i o nc o m m a n d s ,o n ep e rl i n e . E n dw i t hC N T L / Z . N 7 K 1 1 ( c o n f i g ) #i n te 1 / 1-2 ,e 2 / 1-2 N 7 K 1 1 ( c o n f i g i f r a n g e ) #c h a n n e l g r o u p1 0m o d ea c t i v e N 7 K 1 1% E T H _ P O R T _ C H A N N E L 3 C O M P A T _ C H E C K _ F A I L U R E :r a t em o d ei sn o tc o m p a t i b l e c o m m a n df a i l e d :p o r tn o tc o m p a t i b l e[ r a t em o d e ] * *Y o uc a nu s ef o r c eo p t i o nt oo v e r r i d et h ep o r t ' sp a r a m e t e r s * *( e . g ." c h a n n e l g r o u pXf o r c e " ) * *U s e" s h o wp o r t c h a n n e lc o m p a t i b i l i t y p a r a m e t e r s "t og e tm o r ei n f o r m a t i o no nf a i l u r e

After the channels are successfully formed, the s h o wp o r t c h a n n e ls u m m a r y output should indicate that the member links are "Up in the port-channel" with flag (P). This output also shows whether LACP negotiation was used or not.

N 5 K 1 #s h o wp o r t c h a n n e ls u m m a r y F l a g s : D-D o w n s-S u s p e n d e d S-S w i t c h e d P-U pi np o r t c h a n n e l( m e m b e r s ) r-M o d u l e r e m o v e d R-R o u t e d I-I n d i v i d u a l H-H o t s t a n d b y( L A C Po n l y )

U-U p( p o r t c h a n n e l ) M-N o ti nu s e .M i n l i n k sn o tm e t G r o u pP o r t C h a n n e l 1 P o 1 ( S U ) E t h N O N E E t h 1 / 6 ( P ) E t h 1 / 7 ( P ) T y p e P r o t o c o l M e m b e rP o r t s

N 5 K 2 #s h o wp o r t c h a n n e ls u m m a r y F l a g s : D-D o w n s-S u s p e n d e d S-S w i t c h e d P-U pi np o r t c h a n n e l( m e m b e r s ) r-M o d u l e r e m o v e d R-R o u t e d I-I n d i v i d u a l H-H o t s t a n d b y( L A C Po n l y )

U-U p( p o r t c h a n n e l ) M-N o ti nu s e .M i n l i n k sn o tm e t G r o u pP o r t C h a n n e l 4 P o 4 ( S U ) E t h L A C P E t h 1 / 6 ( P ) E t h 1 / 7 ( P ) T y p e P r o t o c o l M e m b e rP o r t s

N 7 K 1 1 #s h o wp o r t c h a n n e ls u m m a r y F l a g s : D-D o w n s-S u s p e n d e d S-S w i t c h e d P-U pi np o r t c h a n n e l( m e m b e r s ) r-M o d u l e r e m o v e d R-R o u t e d I-I n d i v i d u a l H-H o t s t a n d b y( L A C Po n l y )

U-U p( p o r t c h a n n e l ) M-N o ti nu s e .M i n l i n k sn o tm e t G r o u pP o r t C h a n n e l 1 2 3 P o 1 ( S U ) P o 2 ( S U ) P o 3 ( S U ) E t h E t h E t h N O N E L A C P L A C P E t h 2 / 3 ( P ) E t h 1 / 1 ( P ) E t h 2 / 1 ( P ) E t h 2 / 4 ( P ) E t h 1 / 2 ( P ) E t h 2 / 2 ( P ) T y p e P r o t o c o l M e m b e rP o r t s

N 7 K 2 1 #s h o wp o r t c h a n n e ls u m m a r y F l a g s : D-D o w n s-S u s p e n d e d S-S w i t c h e d P-U pi np o r t c h a n n e l( m e m b e r s ) r-M o d u l e r e m o v e d R-R o u t e d I-I n d i v i d u a l H-H o t s t a n d b y( L A C Po n l y )

U-U p( p o r t c h a n n e l ) M-N o ti nu s e .M i n l i n k sn o tm e t G r o u pP o r t C h a n n e l 2 3 4 P o 2 ( S U ) P o 3 ( S U ) P o 4 ( S U ) E t h E t h E t h L A C P L A C P L A C P E t h 1 / 1 ( P ) E t h 2 / 1 ( P ) E t h 2 / 3 ( P ) E t h 1 / 2 ( P ) E t h 2 / 2 ( P ) E t h 2 / 4 ( P ) T y p e P r o t o c o l M e m b e rP o r t s

Spanning-Tree Protocol should see the port channels as one logical link, as shown below. Separate channels that point the same direction in the spanning-tree, such as Port-Channels 2 and 3 below, are still subject to the normal forwarding and blocking rules.

N 7 K 1 1 #s h o ws p a n n i n g t r e ev l a n1 0 V L A N 0 0 1 0 S p a n n i n gt r e ee n a b l e dp r o t o c o lr s t p R o o tI D P r i o r i t y A d d r e s s C o s t P o r t 3 2 7 7 8 0 0 2 6 . 9 8 0 c . 2 1 4 1 1 4 0 9 7( p o r t c h a n n e l 2 )

H e l l oT i m e 2 s e c M a xA g e2 0s e c F o r w a r dD e l a y1 5s e c B r i d g eI D P r i o r i t y A d d r e s s 3 2 7 7 8 ( p r i o r i t y3 2 7 6 8s y s i d e x t1 0 ) 6 8 b d . a b d 7 . 6 0 4 1

H e l l oT i m e 2 s e c M a xA g e2 0s e c F o r w a r dD e l a y1 5s e c I n t e r f a c e P o 1 P o 2 P o 3 D e s gF W D1 R o o tF W D1 A l t nB L K1 1 2 8 . 4 0 9 6P 2 p 1 2 8 . 4 0 9 7P 2 p 1 2 8 . 4 0 9 8P 2 p R o l eS t sC o s t P r i o . N b rT y p e

------

LACP neighbors of N7K2 should see that its System Priority (the first portion of the System ID) has been reduced to a more preferred value of 16384. The output below also shows whether the neighbor is running LACP in active or passive mode, and whether slow or fast LACP hellos are being used.

N 7 K 1 1 #s h o wl a c pn e i g h b o ri n t e r f a c ep o r t c h a n n e l2 F l a g s : S-D e v i c ei ss e n d i n gS l o wL A C P D U s F-D e v i c ei ss e n d i n gF a s tL A C P D U s A-D e v i c ei si nA c t i v em o d e p o r t c h a n n e l 2n e i g h b o r s P a r t n e r ' si n f o r m a t i o n P a r t n e r P o r t E t h 1 / 1 S y s t e mI D P a r t n e r P o r tN u m b e r A g e 2 0 7 7 P a r t n e r F l a g s F A P a r t n e r P o r tS t a t e 0 x 3 f P-D e v i c ei si nP a s s i v em o d e

1 6 3 8 4 , 0 2 6 9 8 c 2 1 4 1 0 x 1 0 1 L A C PP a r t n e r P o r tP r i o r i t y 3 2 7 6 8 P a r t n e r O p e rK e y 0 x 1

P a r t n e r ' si n f o r m a t i o n P a r t n e r P o r t E t h 1 / 2 S y s t e mI D P a r t n e r P o r tN u m b e r A g e 2 0 7 6 P a r t n e r F l a g s F A P a r t n e r P o r tS t a t e 0 x 3 f

1 6 3 8 4 , 0 2 6 9 8 c 2 1 4 1 0 x 1 0 2 L A C PP a r t n e r P o r tP r i o r i t y 3 2 7 6 8 P a r t n e r O p e rK e y 0 x 1

To verify the configured load balancing method of the switches, use the s h o wp o r t c h a n n e ll o a d b a l a n c e command, as seen below.

Note:
Note that on the Nexus 7000 the load balancing method can only be changed in the default VDC, as this change is chassis-wide between all VDCs.

N 5 K 1 #s h o wp o r t c h a n n e ll o a d b a l a n c e P o r tC h a n n e lL o a d B a l a n c i n gC o n f i g u r a t i o n : S y s t e m :s o u r c e d e s t p o r t P o r tC h a n n e lL o a d B a l a n c i n gA d d r e s s e sU s e dP e r P r o t o c o l : N o n I P :s o u r c e d e s t m a c I P :s o u r c e d e s t p o r ts o u r c e d e s t i ps o u r c e d e s t m a c N 7 K 1 #s h o wp o r t c h a n n e ll o a d b a l a n c e P o r tC h a n n e lL o a d B a l a n c i n gC o n f i g u r a t i o n : S y s t e m :s r c d s tl 4 p o r t P o r tC h a n n e lL o a d B a l a n c i n gA d d r e s s e sU s e dP e r P r o t o c o l : N o n I P :s r c d s tm a c I P :s r c d s tl 4 p o r t

To verify load distribution among member ports of the channels in the data plane, Server 1 generates multiple TCP flows to Server 2 using the iPerf testing app.

Verification of the port channel from N5K1 to N7K1 shows a 30-second average output rate of around 750Mbps. This, however, does not tell us which particular member interfaces of the channel are being used.

N 5 K 1 #s h o wi n t e r f a c ep o r t c h a n n e l1|i n c l u d e" o u t p u tr a t e " 3 0s e c o n d so u t p u tr a t e7 5 4 2 6 6 6 3 2b i t s / s e c ,6 1 9 6 6p a c k e t s / s e c i n p u tr a t e3 . 4 0M b p s ,6 . 1 3K p p s ;o u t p u tr a t e7 5 4 . 2 7M b p s ,6 1 . 9 7K p p s

By looking at the individual counters of the member interfaces, we can see that the load is being distributed among them. Note that it is not a 1:1 even distribution, because not every TCP flow is

getting the same about of throughput based on other variables such as burstiness, queueing, etc.

N 5 K 1 #s h o wi n t e r f a c ee 1 / 6-7|i n c l u d e" E t h e r n e t 1 / | o u t p u tr a t e " E t h e r n e t 1 / 6i su p 3 0s e c o n d so u t p u tr a t e4 7 7 7 9 9 7 4 4b i t s / s e c ,3 9 2 5 6p a c k e t s / s e c i n p u tr a t e1 . 6 7M b p s ,3 . 0 1K p p s ;o u t p u tr a t e4 7 7 . 8 0M b p s ,3 9 . 2 6K p p s E t h e r n e t 1 / 7i su p 3 0s e c o n d so u t p u tr a t e2 7 6 4 6 6 8 8 8b i t s / s e c ,2 2 7 1 0p a c k e t s / s e c i n p u tr a t e1 . 7 3M b p s ,3 . 1 2K p p s ;o u t p u tr a t e2 7 6 . 4 7M b p s ,2 2 . 7 1K p p s

^ back to top

Disclaimer (http://www.ine.com/feedback.htm) | Privacy Policy (http://www.ine.com/resources/) Inc., All Rights Reserved (http://www.ine.com/about-us.htm)

2013 INE