I received a rather thoughtful question about the NMNET configuration in Snoqualmie, and though I would take this opportunity

to pass along some information that most people may not be completely aware of (or perhaps care about) related to the 6500 specifically, but generally more or less applies to all L3 switches in some way. Whats an L3 Switch? Pardon the history here, but I think it will help you understand why things are the way they are today As much as 10 years ago, the term L3 Switch was quite the buzzword, causing a mix of emotions across sales and engineering groups, depending on the experiences with the systems. Originally, an L3 switch was one that had some level of routing capability within the chassis, and could not only switch at layer 2, but could also handle the cross-VLAN traffic very cool indeed Originally, long before the idea of VLANs, there would be a routing connecting between switching domains with individual interfaces. Then, when the concept of VLANs could allow logical segmentation within the switch, and more importantly, VLANs could be tagged (or trunked) across a single cable, there would be one big router at the heart of the network, which was responsible for switching all traffic between VLANs. It typically was called a one-armed router because all traffic came and left on the same physical interface, but different logical VLANs within that physical path. This was no different for the 6500. Originally, there was the supervisor module, which had no routing capability at all It was a very fast switch. Then Cisco created a module called a RSM (Routing Switch Module), and later the MSM which actually took a slot in the chassis. It had four virtual Gigabit interfaces, which could be configured as trunks on the switching platform. The MSM had to be configured separately, even though it was within the chassis. It had its own console port, and would be configured with its own IP address to be reached inband. It was literally just a router which used the backplane power, and connected via the backplane to the switching fabric. It was pretty cool, but had some serious limitations and configuration announce. For example, the interfaces actually had to be configured as sub-interfaces, including the Dot1Q commands, to use trunks. If you wanted to use all four Gig ports as a single 4G channel, you had to create a etherchannel/portchannel configuration between the backplane and the MSM. It was VERY cumbersome and difficult to support, but was better than the one-armed router. Next, came the MSFC, which was a very expensive option. It was an integrated RSM/MSM built as a daughter card on the Supervisor card, so you didnt waste a slot. It still was a separate processing unit, but shared the backplane with the supervisor, and therefore had much more bandwidth available to it. There were also some significant improvements to the configuration. For example, the MSFC understood the concept of the VLAN, and was logically separated from the connection with the backplane, since much of that was done in the background. However, much of the same functionality was retained. Today, although you COULD purchase a supervisor without an MSFC, there would be no cost advantage to doing so. Code and Features? When the MSFC first came out, it ran an independent version of code, separate and distinct from the switching supervisor. The two units shared (and still do, actually) the console port, and some of the flash memory. Essentially, if you wanted to configure ports, you configured them on the switch. If you wanted to manipulate routing, it was done on the MSFC, which again, is nothing more than an integrated router with some VLAN intelligence. This original configuration was called a hybrid. When Cisco ships out supervisors, this is the configuration they still come in. For better or worse, this configuration was by far the most simple to support, and has historically been the most reliable in fail-over scenarios. Not until recently has any other configuration been as stable.

Recently (within the last 5 or so years) Cisco decided that supporting CatOS and IOS at the same time on the same device was costing too much, so they chose to move everything to a single code look and feel. So, how do you do that when there are two separate modules running different code? The same way we make any two things talk in the network they share information between themselves behind the scenes. In what is now called Native IOS configurations, the supervisor and MSFC appear to run the same code, and indeed the same file is loaded by both. However, each device runs a very different portion of the code. Both physical modules communicate with each other using the backplane, and running their independent sections of code. Most of this is behind the scenes except when it breaks. 8-D So what is actually happening when you configure things on a Native IOS system? How does an IP Interface work then? One thing that seems to make people very excited about Native IOS on Catalyst systems is that you can have so many routed interfaces. Well, you could have just as many on the Hybrid version, but it actually did require a bit more work, which is now all done behind the scenes. In the Hybrid, you would create a VLAN for each network you wish to have as a routing domain, map a port to it, and assign an IP on the MSFC to that VLAN. The same thing occurs in Native IOS, but most people are not aware of this. To demonstrate this, look at this output, which you may not have seen before. DRCATL01#show vlan internal usage VLAN ---1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1030 1032 1034 1035 1036 1037 1038 Usage -------------------online diag vlan0 online diag vlan1 online diag vlan2 online diag vlan3 online diag vlan4 online diag vlan5 PM vlan process (trunk tagging) L3 multicast partial shortcuts L3 multicast routed port aggregation GigabitEthernet8/1 GigabitEthernet8/2 GigabitEthernet8/14 GigabitEthernet8/15 ATM6/0/0 GigabitEthernet8/3 GigabitEthernet8/4 Serial11/1/0 ATM6/0/0.420 ATM11/0/0 ATM11/0/0.309 ATM11/0/0.401 ATM11/0/0.404 FastEthernet3/1 FastEthernet3/3 FastEthernet3/17 FastEthernet3/20

Now, lets look at which ports are up and routed:

DRCATL01# show interfaces Fa3/1 NET MGMT ISMATL01 10/100BaseTX Fa3/3 To ATSSG001 GIG_ET 10/100BaseTX Fa3/17 104.23_ATSSG02_GE10/100BaseTX Fa3/20 06.05 ATPRAMS1 10/100BaseTX Gi8/1 NET To NRCATL01 G0 1000BaseSX Gi8/2 NET To NRCATL02 G0 1000BaseSX Gi8/3 NET 05.1 CRCATL01_ Gi8/4 NET 05.2 CRCATL02_ Gi8/14 NET 121.07_ORCATL0 1000BaseSX Gi8/15 NET 121.08_ORCATL0 1000Base

status | in connected.*routed connected routed full connected connected connected connected connected connected connected connected connected routed routed routed routed routed routed routed routed routed full full a-full full full full full full full

100 100 100 a-100 1000 1000 1000 1000 1000 1000

All of the above interfaces (and some others not listed by this command, such as WAN interfaces) appear in the list of VLANs. In other words, the switch has assigned VLANs to these ports, and mapped it to a logical (hidden) port, and tied that to a physical port. It is significantly more complex on the backside, but appears simpler to the user. So when you really look under the covers, there is little or no difference in operation between the Native IOS and Hybrid switching functions with regards to routed ports The difference is that in one system, the MSFC is simply a router with only VLAN intelligence, while the other actually uses software to build some of the backend configuration without the user being aware of it. How does this apply to the MSFC trick I did in Snoqualmie? Well, I used the router in the MSFC to route between the VLANs, rather than send the traffic to the FWM. You can get to them by one of several ways 1) SSH to either ISMSNQ01 or ISMSNQ02, then issue session 15 to change VTY console control to the MSFC. Give the ops password, then the standard CORENET Enable password, and you are in. 2) Telnet to any of the IP addresses on any of the VLANs, with the lower address being the *02, and the higher being the *03, and give the same passwords as above. 3) Go through the console port, authenticate using TACACS, then execute switch console to switch the control of the physical console. Switch it back by executing <control-C> three times. There may be other ways, but I cant think of them at this time One very important note The most direct route to the MSFC in Snoqualmie from Atlanta is via the CRMSNQ, which will connect to VLAN321, using either IP 5.196.0.34 (ISMSNQ01) or 5.196.0.35 (ISMSNQ02). If there are other problems on the NMNET network, this is the best way to access the network in Snoqualmie, because it cuts out several hops and routes. Whats the advantage of an L3 switch? Well, there are some advantages Here are a few: 1) The bandwidth between switching planes is much more available 2) There is a lower latency, because fewer physical systems touch the packet

3) The packet can be L2.5 switched, by the PFC (Policy Feature Card) or some other such thing, by building a forwarding cache which will not require examining by the MSFC. 4) Physical Advantages a. Fewer devices b. Less power requirements c. Lower maintenance costs since maintenance is typically by chassis d. Less rack space Other things about the 6500s that you may not know Other cards such as the Flexwan card use the same VLAN features and tricks with VLAN communication across the backplane. In fact, a Flexwan is really nothing more than a pair of 7206 routers with a single processor for each slot. The Flexwan card is a completely separate entity, running independent code. Recall the ATM entries above. Here is some other output you may not have seen or realized. DRCATL01#ipc-con Enter interface slot to connect to: 11 Enter interface cpu to connect to: 0 Entering CONSOLE for slot 11 Type "^C^C^C" to end this session

CWPA-Slot11/0>en CWPA-Slot11/0#show cwan vlan Hidden VLAN DracoMacAddr Interface ------------------------------------------1022 (3FE ) 000b.4568.084a Serial11/1/0 1024 (400 ) 000b.4568.084a ATM11/0/0 1030 (406 ) 000b.4568.084a ATM11/0/0.309 1032 (408 ) 000b.4568.084a ATM11/0/0.401 1034 (40A ) 000b.4568.084a ATM11/0/0.404 ------------------------------------------Deferred VLAN DracoMacAddr Interface CWPA-Slot11/0#dir Directory of bootflash:/ 1 -rw1988122 Nov 26 2002 05:48:53 +00:00 cwpa.bundled

7602176 bytes total (5613924 bytes free) This card runs independent code, which is pushed down by the MSFC during the boot process, if needed. The interfaces communicate with the MSFC via these hidden VLANs. The Flexwan switches switch traffic off the VLAN to the interface, and vice versa. They even have an assigned MAC which appears in the MAC list on the supervisor/MSFC. So, when you look at it based on what is actually going on behind the scenes, it really isnt so complicated And, ultimately it is many pieces bolted onto the original switching fabric of the old CatOS 6500s. So, whether we like it or not, Native IOS is here to stay, along with all the features and problems that come along with it because of all the under the covers tricks it must do to make things easy to configure 8-D