ebgp multipathing with underlying igp load-balancing

Hi!
Today I will spend a little time with a scenario where we are simulating the connection to a service provider and peer with it via eBGP. One special thing here is that we will have some high availability and also load-balancing in the scenario.

Here is our topology:

Goal:
– get the routes from the ISP at both CE routers via eBGP
– be able to load-balance with eBGP to towards the ISPs
– setup an HSRP group for the CE routers to gain HA for the customer routers

So lets draw this down into a few steps:

1)
– configure the ip addressing and loopback interfaces

2)
– configure the HSRP for the customers

3)
– configure the ebgp peering between the ISP1 and CE1

4)
– configure the ebgp peering between the ISP2 and CE2

5)
– configure the ebgp load-balancing between CE1 and ISP1 & ISP2

6)
– optimize the HSRP tracking

Well lets begin:

Step 1)

To configure the IP addressing so that it makes sense I configured the following addresses:

CE1 – ISP1: 172.16.11.0/30 (Where the ISP routers always get the smaller number)
CE2 – ISP2: 172.16.22.0/30
CE1 – CE2: 172.16.21.0/30
CE1/2 – TEST: 192.168.0.0/24 (Those will be the customer routes)
ISP1/2: 100.100.100.0/24, 100.100.101.0/24, 100.100.102.0/24 (ISP routes)

Here are the loopback addresses:

CE1: 1.1.1.1/32
CE2: 2.2.2.2/32
TESTROUTER: not needed
ISP1: 10.10.10.10/32
ISP2: 20.20.20.20/32

Configuration:

CE1(config)#int lo0
CE1(config-if)#ip address 1.1.1.1 255.255.255.255
!
CE1(config-if)#int fa0/0
CE1(config-if)#no shut
CE1(config-if)#ip address 172.16.11.2 255.255.255.252
!
CE1(config-if)#int fa0/1
CE1(config-if)#no shut
CE1(config-if)#ip address 172.16.0.1 255.255.255.252
!
CE1(config)#int fa1/0
CE1(config-if)#no shut
CE1(config-if)#ip address 192.168.0.2 255.255.255.0


CE2(config)#int lo0
CE2(config-if)#ip address 2.2.2.2 255.255.255.255
CE2(config-if)#int fa0/0
CE2(config-if)#no shut
CE2(config-if)#ip address 172.16.22.2 255.255.255.252
!
CE2(config-if)#int fa0/1
CE2(config-if)#no shut
CE2(config-if)#ip address 172.16.0.2 255.255.255.252
!
CE2(config)#int fa1/0
CE2(config-if)#no shut
CE2(config-if)#ip address 192.168.0.3 255.255.255.0

TESTROUTER(config)#int fa0/0
TESTROUTER(config-if)#no shut
TESTROUTER(config-if)#ip address 192.168.0.100 255.255.255.0


ISP1(config)#int lo0
ISP1(config-if)#ip address 10.10.10.10 255.255.255.255
!
ISP1(config-if)#int fa0/0
ISP1(config-if)#ip address 172.16.11.1 255.255.255.252
ISP1(config-if)#no shut
!
ISP1(config-if)#int fa0/1
ISP1(config-if)#ip address 100.100.100.1 255.255.255.0
ISP1(config-if)#ip address 100.100.101.1 255.255.255.0 secondary
ISP1(config-if)#ip address 100.100.102.1 255.255.255.0 secondary
ISP1(config-if)#no shut

ISP2(config)#int lo0
ISP2(config-if)#ip address 20.20.20.20 255.255.255.255
!
ISP2(config-if)#int fa0/0
ISP2(config-if)#ip address 172.16.22.1 255.255.255.252
ISP2(config-if)#no shut
!
ISP2(config-if)#int fa0/1
ISP2(config-if)#ip address 100.100.100.2 255.255.255.0
ISP2(config-if)#ip address 100.100.101.2 255.255.255.0 secondary
ISP2(config-if)#ip address 100.100.102.2 255.255.255.0 secondary
ISP2(config-if)#no shut

 2)
We will now get to a quite simple task. We configure the HSRP group for the CE routers. CE1 will be the primary router and CE2 the failover machine.

CE1(config-if)#standby 100 ip 192.168.0.1
CE1(config-if)#standby 100 priority 150
CE1(config-if)#standby 100 preempt

CE2(config-if)#standby 100 ip 192.168.0.1
CE2(config-if)#standby 100 priority 100
CE2(config-if)#standby 100 preempt

3)
Now its time to configure the ebgp peering between ISP1 and CE1. Usually one does the peering between the physical interfaces but here we want to to load balancing and redundancy so we will use the loopback addresses for the peering as the loopbacks are always up and running.
Due to this we need to use one of two special features. EBGP peerings are usually built between directly connected neighbors. In our case they are not because the loopback addresses are locally to each router and they are /32. Those features are called “EBGP Multihop” which makes it possible to connect neighbors that are up to 255 hops away.
Another feature to do this is the “disable-connected-check” under the BGP neighbor command, but remember that this is only for routers that are directly connected BUT not using the directly connected interfaces. This wont work when one or more routers are inbetween the two potential peers.
As we definitely need EBGP multihop between CE1 and ISP2 later on, we wil also configure the peering between CE1 and ISP1 with that option.

First we need to make sure that the Loopback addresses of ISP1 and CE1 can reach each other. We could use an IGP for that but usually when you peer with ISPs they wont use an IGP as this would be very dangerous and complex to handle with all the filters and so on. So easy option here is “static routing”.

CE1(config)#ip route 1.1.1.1 255.255.255.255 172.16.11.1

ISP1(config)#ip route 10.10.10.10 255.255.255.255 172.16.11.2

Test it:

CE1#ping 10.10.10.10 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/8 ms
CE1#

Looks good! Now we can configure the EBGP peering:

CE1(config)#router bgp 200
CE1(config-router)#neighbor 10.10.10.10 remote-as 100
CE1(config-router)#neighbor 10.10.10.10 ebgp-multihop
CE1(config-router)#neighbor 10.10.10.10 update-source Loopback0

The update-source command here says that we are using Loopback0 to initiate the BGP peering.
On ISP1 respectively:

ISP1(config)#router bgp 100
ISP1(config-router)#neighbor 1.1.1.1 remote-as 200
ISP1(config-router)#neighbor 1.1.1.1 ebgp-multihop
ISP1(config-router)#neighbor 1.1.1.1 update-source Loopback0

The BGP peering now should come up!

CE1#
*Mar 1 00:07:11.779: %BGP-5-ADJCHANGE: neighbor 10.10.10.10 Up

ISP1#
*Mar 1 00:07:14.363: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Up

Now that we have the first peering established we should inject ISP-routes and CUSTOMER-routes to see if that works properly. We will issue the network command in this scenario but route-redistribution would also be good!

IMPORTANT NOTE:
DONT ADVERTISE THE LOOPBACK ADDRESSES INTO EBGP!! WHEN YOU ARE USING AN IGP TO EXCHANGE THEM YOU WILL GET INTO PROBLEMS AS EBGP ADMINISTRATIVE DISTANCE IS THE LOWEST YOU CAN GET IN ROUTING PROTOCOLS. YOU WILL THEN ENCOUNTER THE BGP RACE CONDITION PROBLEM!

CE1(config)#router bgp 200
CE1(config-router)#network 192.168.0.0 mask 255.255.255.0

ISP1(config)#router bgp 100
ISP1(config-router)#network 100.100.100.0 mask 255.255.255.0
ISP1(config-router)#network 100.100.101.0 mask 255.255.255.0
ISP1(config-router)#network 100.100.102.0 mask 255.255.255.0

Check if the routes are being propagated with “sh ip bgp”

ISP1#sh ip bgp
BGP table version is 15, local router ID is 10.10.10.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 100.100.100.0/24 0.0.0.0 0 32768 i
*> 100.100.101.0/24 0.0.0.0 0 32768 i
*> 100.100.102.0/24 0.0.0.0 0 32768 i
*> 192.168.0.0 1.1.1.1 0 0 200 i

CE1#sh ip bgp
BGP table version is 31, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 100.100.100.0/24 10.10.10.10 0 0 100 i
*> 100.100.101.0/24 10.10.10.10 0 0 100 i
*> 100.100.102.0/24 10.10.10.10 0 0 100 i
*> 192.168.0.0 0.0.0.0 0 32768 i

Reachability test from TESTROUTER to see if it is working network wide. Of course TESTROUTER need a default router to the HSRP address of the CE routers:

TESTROUTER(config)#ip route 0.0.0.0 0.0.0.0 192.168.0.1

TESTROUTER#ping 100.100.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.100.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/9/16 ms
TESTROUTER#ping 100.100.101.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.101.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/10/16 ms
TESTROUTER#ping 100.100.102.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.102.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/8 ms
TESTROUTER#

Ok step 3 is now finished.

4)
Now its time to configure the backup machines to come into play when the primary connection is offline.
We establish a peering the same way between CE2 and ISP2 we did for CE1 and ISP1

Again first the static routes for the Loopback reachability:

CE2(config)#ip route 2.2.2.2 255.255.255.255 172.16.22.1

ISP2(config)#ip route 20.20.20.20 255.255.255.255 172.16.22.2

Test it:

CE2#ping 20.20.20.20 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.20.20.20, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
CE2#

BGP Configuration:

CE2(config)#router bgp 200
CE2(config-router)#neighbor 20.20.20.20 remote-as 100
CE2(config-router)#neighbor 20.20.20.20 ebgp-multihop
CE2(config-router)#neighbor 20.20.20.20 update-source Loopback0

ISP2(config)#router bgp 100
ISP2(config-router)#neighbor 2.2.2.2 remote-as 200
ISP2(config-router)#neighbor 2.2.2.2 ebgp-multihop
ISP2(config-router)#neighbor 2.2.2.2 update-source Loopback0

ISP2#
*Mar 1 01:17:20.255: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Up

CE2#
*Mar 1 01:17:18.959: %BGP-5-ADJCHANGE: neighbor 20.20.20.20 Up

Then we need to advertise the networks here as well like we did in our first peering.

CE2(config)#router bgp 200
CE2(config-router)#network 192.168.0.0 mask 255.255.255.0

ISP2(config)#router bgp 100
ISP2(config-router)#network 100.100.100.0 mask 255.255.255.0
ISP2(config-router)#network 100.100.101.0 mask 255.255.255.0
ISP2(config-router)#network 100.100.102.0 mask 255.255.255.0

Now we should also see the routes coming into the BGP table.

CE2#sh ip bgp
BGP table version is 19, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 100.100.100.0/24 20.20.20.20 0 0 100 i
*> 100.100.101.0/24 20.20.20.20 0 0 100 i
*> 100.100.102.0/24 20.20.20.20 0 0 100 i
*> 192.168.0.0 0.0.0.0 0 32768 i

ISP2#sh ip bgp
BGP table version is 16, local router ID is 20.20.20.20
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 100.100.100.0/24 0.0.0.0 0 32768 i
*> 100.100.101.0/24 0.0.0.0 0 32768 i
*> 100.100.102.0/24 0.0.0.0 0 32768 i
*> 192.168.0.0 2.2.2.2 0 0 200 i

Very fine here. So now we wanna test if there is reachability from the TESTROUTER to the 100.100.10X.0 networks. For this we shutdown the Fa1/0 (LAN Interface) of CE1 so that HSRP changes the “active”-role to CE2 and the packets will run over our second peering CE2 – ISP2. Later on we will configure HSRP tracking to manage faults in the WAN as well automatically.

CE1(config)#int fa1/0
CE1(config-if)#shut

CE2#
*Mar 1 01:21:21.431: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 100 state Standby -> Active

We now need to ping the 100.100.10X.2 addresses, as ISP1 does not know where to send the ICMP replys because it has no routes to the 192.168.0.0/24 in its BGP table because the fa1/0 of CE1 is shutdown and the route is no longer advertised to ISP1.

TESTROUTER#ping 100.100.100.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.100.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/8 ms
TESTROUTER#ping 100.100.101.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.101.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/6/8 ms
TESTROUTER#ping 100.100.102.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.102.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/8 ms

Ok this is also working so we can reenable the fa1/0 of CE1

CE1(config)#int fa1/0
CE1(config-if)#no shut

5)
So now we come to the next task. We will now configure load balancing. Load balancing in our case only makes sense for CE1 because when CE1 or ISP1 is down there is no way to load balance accross two routers because there would be only one remaining. As CE1 is the HSRP active router, this one will get a second peering to ISP2.
We will establish that now.

First step, like before, we need to establish end to end connectivity between the Loopbacks of CE1 and ISP2. For this we now need two static routes because the inter-loopback traffic will traverse CE2.

CE1(config)#ip route 20.20.20.20 255.255.255.255 172.16.21.2

ISP2(config)#ip route 1.1.1.1 255.255.255.255. 172.16.22.2

CE2(config)#ip route 1.1.1.1 255.255.255.255 172.16.0.1
CE2(config)#ip route 20.20.20.20 255.255.255.255 172.16.22.1

Connectivity check:

CE1#ping 20.20.20.20 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.20.20.20, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/12 ms

Works! Now we can establish the EBGP peering, with EBGP-multihop again of course.

CE1(config)#router bgp 200
CE1(config-router)#neighbor 20.20.20.20 remote-as 100
CE1(config-router)#neighbor 20.20.20.20 ebgp-multihop
CE1(config-router)#neighbor 20.20.20.20 update-source Loopback0

ISP2(config)#router bgp 100
ISP2(config-router)#neighbor 1.1.1.1 remote-as 200
ISP2(config-router)#neighbor 1.1.1.1 ebgp-multihop
ISP2(config-router)#neighbor 1.1.1.1 update-source Loopback0

CE1(config-router)#
*Mar 1 01:33:45.115: %BGP-5-ADJCHANGE: neighbor 20.20.20.20 Up

ISP2#
*Mar 1 01:33:46.131: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Up

We do not need to enter network commands under the BGP process because we already did. So now on CE1 we should the the ISP routes coming in via ISP1 AND ISP2.

CE1#sh ip bgp
BGP table version is 36, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
* 100.100.100.0/24 20.20.20.20 0 0 100 i
*> 10.10.10.10 0 0 100 i
* 100.100.101.0/24 20.20.20.20 0 0 100 i
*> 10.10.10.10 0 0 100 i
* 100.100.102.0/24 20.20.20.20 0 0 100 i
*> 10.10.10.10 0 0 100 i
*> 192.168.0.0 0.0.0.0 0 32768 i

Here they are. The ones from 10.10.10.10 are marked as best. Lets check why this is the case and take a deeper look.

CE1#sh ip bgp 100.100.100.0
BGP routing table entry for 100.100.100.0/24, version 36
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Flag: 0x800
Advertised to update-groups:
1
100
20.20.20.20 from 20.20.20.20 (20.20.20.20)
Origin IGP, metric 0, localpref 100, valid, external
100
10.10.10.10 from 10.10.10.10 (10.10.10.10)
Origin IGP, metric 0, localpref 100, valid, external, best

Well they are equal…ALMOST :). From the BGP path selection process we can refer to Step 9 – OLDEST ROUTE FIRST. As all bgp attributes are equal the route that came first will win here. And remember we established the peering to 10.10.10.10 before the peering to 20.20.20.20. We can try this out by shutting down the neighbor 10.10.10.10 and reenable it. Then the path to 20.20.20.20 should be preferred.

CE1(config)#router bgp 200
CE1(config-router)#neighbor 10.10.10.10 shut
CE1(config-router)#
*Mar 1 01:38:04.367: %BGP-5-ADJCHANGE: neighbor 10.10.10.10 Down Admin. shutdown

CE1#sh ip bgp 100.100.100.0
BGP routing table entry for 100.100.100.0/24, version 37
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x800
Advertised to update-groups:
1
100
20.20.20.20 from 20.20.20.20 (20.20.20.20)
Origin IGP, metric 0, localpref 100, valid, external, best

Of course only one entry in the BGP table. We now reenable the peering to 10.10.10.10 and see what happens.

CE1(config)#router bgp 200
CE1(config-router)#no neighbor 10.10.10.10 shut
CE1(config-router)#
*Mar 1 01:39:04.399: %BGP-5-ADJCHANGE: neighbor 10.10.10.10 Up

CE1#sh ip bgp 100.100.100.0
BGP routing table entry for 100.100.100.0/24, version 42
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Flag: 0x800
Advertised to update-groups:
1
100
10.10.10.10 from 10.10.10.10 (10.10.10.10)
Origin IGP, metric 0, localpref 100, valid, external
100
20.20.20.20 from 20.20.20.20 (20.20.20.20)
Origin IGP, metric 0, localpref 100, valid, external, best

Voila! we were right :). But we still got a problem. We wanted to do load-balancing. But BGP per defualt only allows ONE path from the BGP table (RIB, routing infomration base) to be installed into the FIB (routing-table).

CE1#sh ip route 100.100.100.0
Routing entry for 100.100.100.0/24
Known via "bgp 200", distance 20, metric 0
Tag 100, type external
Last update from 20.20.20.20 00:00:18 ago
Routing Descriptor Blocks:
* 20.20.20.20, from 20.20.20.20, 00:09:31 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 100

We need to modify the BGP process a little and use the “maximum-paths” command under the BGP process.

CE1(config)#router bgp 200
CE1(config-router)#maximum-paths 2

CE1#sh ip bgp 100.100.100.0
BGP routing table entry for 100.100.100.0/24, version 52
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Multipath: eBGP
Flag: 0x800
Advertised to update-groups:
1
100
10.10.10.10 from 10.10.10.10 (10.10.10.10)
Origin IGP, metric 0, localpref 100, valid, external, multipath
100
20.20.20.20 from 20.20.20.20 (20.20.20.20)
Origin IGP, metric 0, localpref 100, valid, external, multipath, best

Now we can see there is more information…the keyword “multipath” comes up! Lets check the FIB!

CE1#sh ip route 100.100.100.0
Routing entry for 100.100.100.0/24
Known via "bgp 200", distance 20, metric 0
Tag 100, type external
Last update from 10.10.10.10 00:02:09 ago
Routing Descriptor Blocks:
* 20.20.20.20, from 20.20.20.20, 00:12:31 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 100
10.10.10.10, from 10.10.10.10, 00:02:09 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 100

Beauty!!! 🙂 We are now load-balancing!
BUT…wait a moment lets have a look at the BGP output again.

CE1#sh ip bgp 100.100.100.0
BGP routing table entry for 100.100.100.0/24, version 52
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Multipath: eBGP
Flag: 0x800
Advertised to update-groups:
1
100
10.10.10.10 from 10.10.10.10 (10.10.10.10)
Origin IGP, metric 0, localpref 100, valid, external, multipath
100
20.20.20.20 from 20.20.20.20 (20.20.20.20)
Origin IGP, metric 0, localpref 100, valid, external, multipath, best

There is only one route that is marked “best” although both ones are in the routing table. Well obviously the router IS load-balancing because thats what the routing-table says us. How can we see if the packets are really load-balanced?
Load-Balancing generally is based on a “per destination” basis. So lets generate some traffic to different destinations from the TESTROUTER.

TESTROUTER#ping 100.100.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.100.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 8/16/32 ms
TESTROUTER#ping 100.100.101.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.101.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 8/16/32 ms
TESTROUTER#ping 100.100.102.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.100.102.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/12 ms
TESTROUTER#

Okay here is a command where we can see where the packets are really going. We know that the packets are 100% routed through CE1 because he is the HSRP active router.
So we have the following 3 DIFFERENT flows now:

ICMP: FROM 192.168.0.100 -> 100.100.100.1
ICMP: FROM 192.168.0.100 -> 100.100.101.1
ICMP: FROM 192.168.0.100 -> 100.100.102.1

On CE1 we enter a very useful command: “show ip cef exact-route” . This command enables us to see where the packets are really going. Lets have a look.

CE1#sh ip cef exact-route 192.168.0.100 100.100.101.2
192.168.0.100 -> 100.100.101.2 : FastEthernet0/0 (next hop 172.16.11.1)
CE1#sh ip cef exact-route 192.168.0.100 100.100.102.2
192.168.0.100 -> 100.100.102.2 : FastEthernet0/0 (next hop 172.16.11.1)
CE1#sh ip cef exact-route 192.168.0.100 100.100.100.2
192.168.0.100 -> 100.100.100.2 : FastEthernet0/1 (next hop 172.16.0.2)

And there we are! Some packets are sent through CE2 to ISP2 and some to ISP1 directly! So we are really load-balancing!

6)
So now the last part is some HSRP tracking. We will take a look on the following disaster scenarios.

a) CE1 crashes – Nothing to, because HSRP will force CE2 to take over and everthing will continue forwarding
b) ISP1 crashes – We will track the interface line-protocol and the Loopback address as there maybe is only a problem in the control-plane of ISP1 and the link stays up, which would be bad.
c) Misconfiguration in inbound BGP filtering to our neighbors – maybe there is a configuration error or a problem with the BGP process of our router. In this example we configure a route-map to let in the prefixes of our neighbors with a mistyped configuration.

b)
First of all we could use under our Fa1/0 Interface of CE1 the command “standby 100 track INTERFACE XX”. But instead of this we will use a “track” in combination with “ip sla” to do this as we need to combine the tracking of the line-protocol AND the loopback-address of ISP1.

First: Check the line Protocol of the WAN Inteface to ISP1

CE1(config)#track 1 interface fa0/0 line-protocol

Second: Preidically check the loopback of ISP1 every 2 seconds for reachability:

CE1(config)#ip sla 2
CE1(config-ip-sla)#icmp-echo 10.10.10.10 source-interface lo0
CE1(config-ip-sla-echo)#timeout 1000
CE1(config-ip-sla-echo)#freq
CE1(config-ip-sla-echo)#frequency 2
!
CE1(config)#track 2 rtr 2 reachability
CE1(config)#ip sla schedule 2 start-time now life forever

*Mar 1 03:07:58.807: %TRACKING-5-STATE: 2 rtr 2 reachability Down->Up

Check the tracks:

CE1#sh track
Track 1
Interface FastEthernet0/0 line-protocol
Line protocol is Up
1 change, last change 00:03:51
Track 2
Response Time Reporter 2 reachability
Reachability is Up
2 changes, last change 00:01:27
Latest operation return code: OK
Latest RTT (millisecs) 16

Now we need to combine the tracks that either the loopback goes down or the line-protocol goes down (for this we need a BOOLEAN AND), HSRP does a switchover:

CE1(config)#track 100 list boolean and
CE1(config-track)#object 1
CE1(config-track)#object 2

*Mar 1 03:10:50.379: %TRACKING-5-STATE: 100 list boolean or Down->Up

CE1#sh track 100
Track 100
List boolean or
Boolean AND is Up
2 changes, last change 00:00:30
object 1 Up
object 2 Up

We can see that the track should go down once one of the objects (the other two tracks we configured) go down. Lets shutdown loopback0 of ISP1.

ISP1(config)#int lo0
ISP1(config-if)#shut

CE1#
*Mar 1 03:55:13.863: %TRACKING-5-STATE: 2 rtr 2 reachability Up->Down
*Mar 1 03:55:14.663: %TRACKING-5-STATE: 100 list boolean and Up->Down

CE1#sh track 100
Track 100
List boolean and
Boolean AND is Down
5 changes, last change 00:00:06
object 1 Up
object 2 Down

Ok that works, now we go to step C.

c)
Additionally we now want to track the routes we get from our BGP peers. We can write a track that checks the presence of routes in the routing table.

CE1(config)#track 10 ip route 100.100.100.0/24 reachability
CE1(config)#track 11 ip route 100.100.101.0/24 reachability
CE1(config)#track 12 ip route 100.100.102.0/24 reachability


CE1#sh track 100
*Mar 1 03:59:18.995: %SYS-5-CONFIG_I: Configured from console by console
CE1#sh track 10
Track 10
IP route 100.100.100.0 255.255.255.0 reachability
Reachability is Up (BGP)
1 change, last change 00:00:25
First-hop interface is FastEthernet0/1
CE1#sh track 11
Track 11
IP route 100.100.101.0 255.255.255.0 reachability
Reachability is Up (BGP)
1 change, last change 00:00:19
First-hop interface is FastEthernet0/1
CE1#sh track 12
Track 12
IP route 100.100.102.0 255.255.255.0 reachability
Reachability is Up (BGP)
1 change, last change 00:00:17
First-hop interface is FastEthernet0/1

We just now need to add them to the boolean list. Once one route fails, the switchover will happen.

CE1(config)#track 100 list boolean and
CE1(config-track)#object 10
CE1(config-track)#object 11
CE1(config-track)#object 12

CE1#sh track 100
Track 100
List boolean and
Boolean AND is Up
6 changes, last change 00:00:03
object 1 Up
object 2 Up
object 10 Up
object 11 Up
object 12 Up

So for now everthing is fine. We will now setup a route-map which ensures that we only get routes for the 100.100.10X.0 networks.

CE1(config)#ip prefix-list PFX-BGP-IN seq 10 permit 100.100.100.0/24
CE1(config)#ip prefix-list PFX-BGP-IN seq 20 permit 100.100.101.0/24
CE1(config)#ip prefix-list PFX-BGP-IN seq 30 permit 100.100.102.0/24
!
CE1(config)#route-map RMAP-BGP-IN permit 10
CE1(config-route-map)#match ip address prefix-list PFX-BGP-IN
CE1(config-route-map)#route-map RMAP-BGP-IN deny 20
!
CE1(config)#router bgp 200
CE1(config-router)#neighbor 10.10.10.10 route-map RMAP-BGP-IN in
CE1(config-router)#neighbor 20.20.20.20 route-map RMAP-BGP-IN in
CE1(config-router)#do clear ip bgp *

The restart of the BGP process is necessary everytime a BGP config change has been made. Its not when BGP peers support BGP route-refresh aka BGP soft inbound reconfiguration.

OUr route tracks are fine for now:

*Mar 1 04:08:40.194: %BGP-5-ADJCHANGE: neighbor 10.10.10.10 Up
*Mar 1 04:08:40.206: %BGP-5-ADJCHANGE: neighbor 20.20.20.20 Up
*Mar 1 04:08:41.142: %TRACKING-5-STATE: 10 ip route 100.100.100.0/24 reachability Down->Up
*Mar 1 04:08:41.142: %TRACKING-5-STATE: 11 ip route 100.100.101.0/24 reachability Down->Up
*Mar 1 04:08:41.142: %TRACKING-5-STATE: 12 ip route 100.100.102.0/24 reachability Down->Up

We will modify the prefix-list with an error for the 100.100.101.0 network.

CE1(config)#no ip prefix-list PFX-BGP-IN seq 20 permit 100.100.101.0/24
CE1(config)#ip prefix-list PFX-BGP-IN seq 20 permit 100.200.101.0/24
!
CE1(config)#do clear ip bgp *

And there we are:

*Mar 1 04:11:01.478: %BGP-5-ADJCHANGE: neighbor 10.10.10.10 Up
*Mar 1 04:11:01.490: %BGP-5-ADJCHANGE: neighbor 20.20.20.20 Up
CE1(config)#
*Mar 1 04:11:11.146: %TRACKING-5-STATE: 11 ip route 100.100.101.0/24 reachability Up->Down
*Mar 1 04:11:11.758: %TRACKING-5-STATE: 100 list boolean and Up->Down

We have our disaster :).

So finally what we need to do after the testing is bind the HSRP process to our combined track 100 and correct the prefix-list.

CE1(config)#int fa1/0
CE1(config-if)#standby 100 track 100 decrement 75
*Mar 1 04:12:27.814: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 100 state Active -> Speak
*Mar 1 04:12:37.814: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 100 state Speak -> Standby

CE1(config)#no ip prefix-list PFX-BGP-IN seq 20 permit 100.200.101.0/24
CE1(config)#ip prefix-list PFX-BGP-IN seq 20 permit 100.100.101.0/24
CE1(config)#do clear ip bgp *

And everything is good again!

*Mar 1 04:14:11.166: %TRACKING-5-STATE: 10 ip route 100.100.100.0/24 reachability Down->Up
*Mar 1 04:14:11.166: %TRACKING-5-STATE: 11 ip route 100.100.101.0/24 reachability Down->Up
*Mar 1 04:14:11.166: %TRACKING-5-STATE: 12 ip route 100.100.102.0/24 reachability Down->Up
*Mar 1 04:14:11.826: %TRACKING-5-STATE: 100 list boolean and Down->Up
*Mar 1 04:14:12.834: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 100 state Standby -> Active

I hope you enjoyed it! Feel free to comment!
Regards!
Markus

Advertisements

About markus.wirth

Living near Limburg in Germany, working as a Network Engineer around Frankfurt am Main.
This entry was posted in BGP and tagged , , , , , , , , , , , , , , , , , . Bookmark the permalink.

One Response to ebgp multipathing with underlying igp load-balancing

  1. raja says:

    excellent article.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s