debugging: part 1 – basic debugging

Packet debugging: Part I – basic debugging

Hi everybody!
This session describes how to do packet debugging within real gear and simulated environemt. I will show this within some (not all) situations where debugging can be useful and where one should definitely not use it without filters.

First of all there are many debug commands under a cisco machine. I used a very simple topology with two neighboring routers to demonstrate some of the basic debugging functions.

Im my case I used IOS Version 12.4(15)T14 with the ADV.ENT feature set. If you use a different IOS version I cannot guarantee that my output will match 100% but in the most cases this should fit.

IMPORTANT NOTE:
Debugging on cisco routers takes a lot of CPU power as it is done in software. So take care what you debug and which filter you use. I will show you how to apply debug filters and so on in the next few parts. First of all to limit the debug output you can rate-limit it. If you have a testmachine then you could do “debug all”. Have fun with it because your machine will get stuck within some seconds as the CPU cant handle the debug output.
To prevent yourself from getting too much output you can use the Route(config)#debug rate-limit command and choose how many messages you want to get per second. But keep in mind that you also will miss some debug messages that might let you dont see what you should see.

“DEBUG IP PACKET”

So the first command I would like to introduce is the “debug ip packet”-command. That command (well the name says it already) show debug output of IP packets. Short story long here is a sample output of the command in our 2-router topology withing dynamips.

This output is shown when R1 sends an IP ICMP packet from hist fa0/0 to the fa0/0 of R2.

R1#debug ip packet
IP packet debugging is on
R1#ping 172.16.21.2 rep 1


Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 172.16.21.2, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 36/36/36 ms
R1#
*Mar 1 21:28:33.482: IP: tableid=0, s=172.16.21.1 (local), d=172.16.21.2 (FastEthernet0/0), routed via FIB
*Mar 1 21:28:33.486: IP: s=172.16.21.1 (local), d=172.16.21.2 (FastEthernet0/0), len 100, sending
*Mar 1 21:28:33.514: IP: tableid=0, s=172.16.21.2 (FastEthernet0/0), d=172.16.21.1 (FastEthernet0/0), routed via RIB
*Mar 1 21:28:33.514: IP: s=172.16.21.2 (FastEthernet0/0), d=172.16.21.1 (FastEthernet0/0), len 100, rcvd 3

As you can see the ICMP has been generated and the router has an entry with the source and destination of the underlying IP packet.
Source is local to the router as it is his own interface and the destination is the remote router (R2). The destination interface which is here FastEthernet0/0 of R1 means that the R1´s way to R2 goes via Fa0/0. Primarily one could say “no thats not right because R1´s way to R2 is via 172.16.21.2”, but there is a mechanism called CEF (cisco express forwarding) that generates shortcuts for the router. The router dynamically creates shortcuts that contain a destination network and the corresponding exit interface for that destination.

R1#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 drop Null0 (default route handler entry)
0.0.0.0/32 receive
172.16.21.0/30 attached FastEthernet0/0
172.16.21.0/32 receive
172.16.21.1/32 receive
172.16.21.2/32 172.16.21.2 FastEthernet0/0
172.16.21.3/32 receive
224.0.0.0/4 drop
224.0.0.0/24 receive
255.255.255.255/32 receive


R1#sh ip cef 172.16.21.2
172.16.21.2/32, version 11, epoch 0, connected, cached adjacency 172.16.21.2
0 packets, 0 bytes
via 172.16.21.2, FastEthernet0/0, 0 dependencies
next hop 172.16.21.2, FastEthernet0/0
valid cached adjacency

This CEF table is called “FIB – Forward Information Base”.

“DEBUG IP ICMP”

Another debug one can do is the “debug ip icmp”.
That debug is useful for checking icmp packets and icmp unreachables. Lets try this. We enable it and send a ping from one router to the other.

R1#ping 172.16.21.2


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.21.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/24/40 ms

*Mar 1 00:14:26.151: ICMP: echo reply sent, src 172.16.21.2, dst 172.16.21.1
*Mar 1 00:14:26.195: ICMP: echo reply sent, src 172.16.21.2, dst 172.16.21.1
*Mar 1 00:14:26.227: ICMP: echo reply sent, src 172.16.21.2, dst 172.16.21.1
*Mar 1 00:14:26.243: ICMP: echo reply sent, src 172.16.21.2, dst 172.16.21.1
*Mar 1 00:14:26.255: ICMP: echo reply sent, src 172.16.21.2, dst 172.16.21.1

What we can also do is, as already mentioned, use that command for seeing icmp unreachables. ICMP unreachables are sent when a router does not have a router to a specific destination and tells the source of the packet that the destination is unreachable (by the way this feature can be disabled so that the source will have no clue wether its a routing problem or anything else). To provocate that we can add a static route to R1 with a destination that does not exist in R2´s routing-table.

R1#conf t
R1(config)#ip route 10.10.10.10 255.255.255.255 172.16.21.2


R2#debug ip icmp
ICMP packet debugging is on

R1#ping 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
U.U.U
Success rate is 0 percent (0/5)

You can see within the ICMP requests you get an “U”. That means “unreachable” lets see what the icmp debug says.

R2#
*Mar 1 04:42:03.902: ICMP: dst (10.10.10.10) host unreachable sent to 172.16.21.1

Here we go. Also a good info that ICMP debugging can be useful. Lets try to disable this feature for testing purposes.

R2#conf t´
R2(config)#int fa0/0
R2(config-if)#no ip unreachables

Lets ping again and we see that we dont get unreachables:

R1#ping 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

There is no debug output on R2. We would need to use “debug ip packet” as ICMP unreachables are blocked from the CPU/CLI/debug due to the interface configuration of the “no ip unreachables”

R2#debug ip packet detail
IP packet debugging is on (detailed)

Again we dont get any debug output…but why??? Well thats because of CEF. CEF means hardware based forwarding and although GNS/dynamips is completely emulated in software it anyways behaves like the hardware CEF. We need to disable CEF only on the interface fa0/0 to force the packets to get “process switched” (switched/routed in software). This has the effect that all packets must run through the software and can be debugged. If you dont get any debug output and the traffic you want to debug is not destined for the itself you are debugging on, then you need to disable CEF all over the machine or on a particular interface.

IMPORTANT NOTE:
In production networks I can HIGLY RECOMMEND NOT TO DISABLE CEF!!! If there are multiple 100mbit/s or even gbit/s running through the machine and you disable CEF then you can go for a walk into the data center to power cycle the device or you are lucky and the device reboots itsefl after it crashed.

CEF can be disabled globally by entering “no ip cef”.
CEF can be disabled on an interface by entering “no ip route-cache” under the interface.

You can check the status of the CEF with that show command:

R2#sh ip int fa0/0
FastEthernet0/0 is up, line protocol is up
Internet address is 172.16.21.2/30
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF Fast switching turbo vector
IP multicast fast switching is disabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled

Lets disable CEF and see what happens:

R2#conf t
R2(config)#int fa0/0
R2(config-if)#no ip route-cache

R2#sh ip int fa0/0 | inc CEF
IP CEF switching is disabled
IP route-cache flags are No CEF

Lets try th eping again:

R1#ping 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R2#
*Mar 1 04:58:21.914: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, unroutable
*Mar 1 04:58:21.918: ICMP type=8, code=0
R2#
*Mar 1 04:58:23.902: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, unroutable
*Mar 1 04:58:23.902: ICMP type=8, code=0
R2#
*Mar 1 04:58:25.902: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, unroutable
*Mar 1 04:58:25.902: ICMP type=8, code=0
R2#
*Mar 1 04:58:27.902: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, unroutable
*Mar 1 04:58:27.902: ICMP type=8, code=0
R2#
*Mar 1 04:58:29.930: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, unroutable
*Mar 1 04:58:29.930: ICMP type=8, code=0

We can suddenly see the packets and we can see that they are unroutable as the router itself does not have any route or exit-interface for that destination. The ICMP type=8 means that this packet is an “ICMP Echo Request”. A list of ICMP code types you can find here -> https://chasingmyccie.wordpress.com/2012/05/08/reminder-icmp-type-codes/

R2#conf t
R2(config)#int fa0/0
R2(config-if)#ip unreachables

Now we should also see at least one more ICMP packet that should equal “type 3”

R2#
*Mar 1 05:26:36.826: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, unroutable
*Mar 1 05:26:36.826: ICMP type=8, code=0
*Mar 1 05:26:36.826: ICMP: dst (10.10.10.10) host unreachable sent to 172.16.21.1
*Mar 1 05:26:36.830: IP: tableid=0, s=172.16.21.2 (local), d=172.16.21.1 (FastEthernet0/0), routed via FIB
*Mar 1 05:26:36.830: IP: s=172.16.21.2 (local), d=172.16.21.1 (FastEthernet0/0), len 56, sending
*Mar 1 05:26:36.830: ICMP type=3, code=1

What we now can do to solve that issue here is to generate a loopback interface with the address 10.10.10.10/32. R2 should then send an ICMP type=0 back.

R2(config)#int lo0
R2(config-if)#ip address 10
*Mar 1 05:31:34.670: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
R2(config-if)#ip address 10.10.10.10 255.255.255.255

R1#ping 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/36/76 ms

R2(config-if)#
*Mar 1 05:32:16.006: IP: tableid=0, s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10 (Loopback0), routed via RIB
*Mar 1 05:32:16.006: IP: s=172.16.21.1 (FastEthernet0/0), d=10.10.10.10, len 100, rcvd 4
*Mar 1 05:32:16.006: ICMP type=8, code=0
*Mar 1 05:32:16.010: ICMP: echo reply sent, src 10.10.10.10, dst 172.16.21.1
*Mar 1 05:32:16.010: IP: tableid=0, s=10.10.10.10 (local), d=172.16.21.1 (FastEthernet0/0), routed via FIB
*Mar 1 05:32:16.010: IP: s=10.10.10.10 (local), d=172.16.21.1 (FastEthernet0/0), len 100, sending
*Mar 1 05:32:16.014: ICMP type=0, code=0

And it works as designed!

Advertisements

About markus.wirth

Living near Limburg in Germany, working as a Network Engineer around Frankfurt am Main.
This entry was posted in Debugging. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s