This post goes through how to configure MPLS VPN on the service-side of a Cisco SD-WAN edge device, so the south-side towards non-SD-WAN devices. What I am trying to achieve is to advertise the differing SD-WAN VPN (VRF, why Cisco have to call these VPNs beats me) prefixes to a core switch (using a ASR in the lab) directly connected to the SD-WAN router. This could be accomplished using per-VRF interfaces (or sub-interfaces) and BGP peerings, but a neater solution is to pass all the routing information over one the BGP MPLS VPNv4 peering.
So lets start with the 17.2.1r release notes on the matter.
The Cisco IOS XE SD-WAN devices support Multiprotocol Label Switching (MPLS) to enable multiple protocol environment. MPLS offers an extremely scalable, protocol agnostic, data-carrying mechanism that transfers data packets with assigned labels across the network through virtual links. Extensions of the BGP protocol can be used to manage an MPLS path. The Cisco IOS XE SD-WAN devices also have the capability of BGP MPLS VPN Option B.
The multiple service VPNs use inter autonomous system (AS) BGP labeled path to forward the traffic, that in turn helps scale the service side VPNs with less control plane signaling. MPLS interface is supported only in global VRF.
|MPLS-BGP Support on the Service Side||Cisco IOS XE Release 17.2.1r||This features allows you to enable support on Multiprotocol Label Switching (MPLS). Multiple Service VPNs use inter autonomous system (AS) BGP labelled path to forward the traffic, which in turn helps scaling the service side VPNs with less control plane signaling. Label distribution for a given VPN routing and forwarding (VRF) instance on a given device can be handled by Border Gateway Protocol (BGP).|
As you can see the information is not really the greatest, the biggest point I took from this is ‘MPLS interface is supported only in global VRF’ which means it has to be configured under the transport VPN as service-side VPNs are not in the global VRF.
Is probably worth a quick recap of MPLS VPN before moving on to the configuration. There are three options to establish an Inter-AS:
- Option A - Back-to-Back VRFs: There is no end-to-end LSP (label switched path) as although LSP may be used for Intra-AS connections to devices within a site (behind the edge router), packets are sent unlabelled between sites (Inter-AS) on a per-VRF basis using IGP or IPv4 BGP.
- Option B - BGP IPv4 and VPNv4: The link between edge routers has no IGP or LDP configuration as instead the inter-AS connection uses the IPv4 and VPNv4 BGP address-families to generate and exchange labels in the BGP updates between neighbors.
- Option C - BGP VPNv4 with LDP: Between edge routers LDP is used to form neighbors and communicate label information whilst IGP is used advertise loopbacks which then in turn form the eBGP VPNv4 peerings to exchange the prefix information.
Both option B and C use the BGP VPNv4 BGP address family to get the routing information between sites and to tell the remote router what label value to use. Option B is exchanging these labels using an IPv4 BGP address-family peering rather than LDP. This blog by Daniel Dib offers a really clear explanation of the differences between the options.
Configure the SD-WAN Edge device
Although the SD-WAN MPLS VPN BGP peering is going to carry service-side traffic the MPLS interface must be in the global routing table which means the interface and BGP feature templates are added under the transport and management VPN section of the device template.
Create a VPN Interface Ethernet template: The configuration of the feature template is no different to any other service-side interface. tunnel mode should remain disabled as although it is associated under the transport VPN it is not actually a transport interface.
Create a BGP template: Within the template define the MPLS interface that the BGP peering will be established over (created in step 1) aswell as the BGP neighbour.
The address family is VPNv4-unicast, all other settings can be left as default. Do not try and set any redistribution or RD/RT within the template, this is done under the VRF’s BGP address family.
- Update the device template: Add the interface and BGP template to the transport and management VPN section of the device template.
For any old school CLI people like myself, the resulting configuration that this creates is as follows:
GigabitEthernet4description MPLS VPN connection no shutdown arp timeout 1200 mpls bgp forwarding ip address 192.168.20.2 255.255.255.240no ip redirects ip mtu 1500 load-interval 30 mtu 1500 !mpls label mode all-vrfs protocol bgp-vpnv4 per-vrfmpls label mode all-vrfs protocol bgp-vpnv6 per-vrfmpls label range 100000 1048575 static 16 99 !router bgp 65106bgp router-id 192.168.20.2neighbor 192.168.20.1remote-as 65106neighbor 192.168.20.1description DC0-ASR-LAB01 MPLS BGP sessionneighbor 192.168.20.1ebgp-multihop 1 neighbor 192.168.20.1send-community both address-family vpnv4 unicast neighbor 192.168.20.1activate neighbor 192.168.20.1send-community both
At this point the BGP peering will be up but nothing learnt or advertised until the service-side VPN is added.
Add service-side VPN templates: Create a VPN template for the VRF, a VPN Interface Ethernet template for the loopback (used for testing) and a VRF specific BGP template (address-family). The BGP template advertises the loopback interface, redistributes OMP routes from this VPN into BGP and set the import route-target to import prefixes learnt from the remote router.
Apply service-side VPN templates: Add the BGP and loopback interface feature templates as sub-templates under the service-side VPN within the device template.
The resulting configuration of this is as follows:
omp address-family ipv4 vrf
40advertise bgp !vrf definition 40description GRYrd 1:40address-family ipv4 route-target export 65106:40route-target import 65106:3006route-target import 65106:40 !interface Loopback10no shutdown arp timeout 1200 vrf forwarding 40ip address 22.214.171.124 255.255.255.255no ip redirects ip mtu 1500 !router bgp 65106address-family ipv4 unicast vrf 40bgp router-id 126.96.36.199network 188.8.131.52 mask 255.255.255.255redistribute omp
Configure neighboring device
For the purpose of this demo I am using a Cisco ASR1002X.
- Configure connecting interface: The interface that connects to the SD-WAN router is in the global routing table and must have mpls bgp forwarding enabled. I am only using sub-interface on port-channel as this equipment is part of a bigger lab, this could be a normal interface.
Port-channel11.3020encapsulation dot1Q 3020ip address 192.168.20.1 255.255.255.240mpls bgp forwarding
- Add the VRF: The RT import values should match the RT export values on the SD-WAN router.
GRYrd 192.168.255.61:3006address-family ipv4 route-target export 65106:3006route-target import 65106:40
- Create IPv4 and VPNv4 BGP peerings: Needs an IPv4 and VPNv4 BGP peering between devices to allow for the sending of the MPLS labels with the BGP routes to the neighbor. LDP won’t assign a label for all the prefixes, it will only assign labels for next-hops.
65106bgp router-id 192.168.255.61no bgp default ipv4-unicastneighbor 192.168.20.2remote-as 65106neighbor 192.168.20.2timers 3 9 address-family ipv4 neighbor 192.168.20.2activate neighbor 192.168.20.2send-labeladdress-family vpnv4 import l2vpn evpn re-originate neighbor 192.168.20.2activate neighbor 192.168.20.2send-community both neighbor 192.168.20.2route-reflector-client neighbor 192.168.20.2next-hop-self
- Advertised loopback: A test loopback is added in the VRF and advertised to the neighboring SD-WAN router.
Loopback10vrf forwarding GRYip address 184.108.40.206 255.255.255.255 !router bgp 65106address-family ipv4 vrf GRYnetwork 220.127.116.11mask 255.255.255.255
The IPv4 and VPNv4 peerings are up with the VRF specific prefixes learnt over the VPNv4 peering.
DC0-SDWAN-LAB01#show ip bgp summary BGP router identifier 192.168.20.2, local AS number 65106 BGP table version is 1, main routing table version 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.20.1 4 65106 1053 985 1 0 0 00:49:45 0 DC0-SDWAN-LAB01#show bgp vpnv4 unicast all summary BGP router identifier 192.168.20.2, local AS number 65106 BGP table version is 25, main routing table version 25 8 network entries using 2048 bytes of memory 8 path entries using 1088 bytes of memory 5/5 BGP path/bestpath attribute entries using 1520 bytes of memory 3 BGP extended community entries using 104 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 4760 total bytes of memory BGP activity 29/21 prefixes, 57/49 paths, scan interval 60 secs 8 networks peaked at 22:08:48 Feb 15 2022 BST (01:36:12.801 ago) Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.20.1 4 65106 1054 986 25 0 0 00:49:47 3
There is reachability to the loopbacks from either device.
DC0-SDWAN-LAB01#ping vrf 40 18.104.22.168 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 22.214.171.124, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms DC0-ASR-LAB01#ping vrf GRY 126.96.36.199 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 188.8.131.52, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
When looking at the labels within the BGP updates you will see two possible labels (mpls labels in/out):
- The inner label identifies the local egress VRF
- The outer label identifies the egress VRF on the neighbouring device
On the SD-WAN router we can see the inner label for the local loopback address (184.108.40.206) is 100000.
DC0-SDWAN-LAB01#show bgp vrf 40 vpnv4 unicast 220.127.116.11 BGP routing table entry for 1:40:18.104.22.168/32, version 18 Paths: (1 available, best #1, table 40) Advertised to update-groups: 9 Refresh Epoch 1 Local 0.0.0.0 (via vrf 40) from 0.0.0.0 (22.214.171.124) Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best Extended Community: RT:65106:40 mpls labels in/out IPv4 VRF Aggr:100000/nolabel(40) rx pathid: 0, tx pathid: 0x0 Updated on Feb 15 2022 23:32:44 BST DC0-SDWAN-LAB01#show mpls forwarding-table vrf 40 Local Outgoing Prefix Bytes Label Outgoing Next Hop Label Label or Tunnel Id Switched interface 100000 No Label IPv4 VRF[V] 1633083 aggregate/40
The loopback learnt from the neighbouring router (126.96.36.199) has an outer label of 97 that is received via BGP.
DC0-SDWAN-LAB01#show ip route vrf 40 188.8.131.52 Routing Table: 40 Routing entry for 184.108.40.206/32 Known via "bgp 65106", distance 200, metric 0, type internal Redistributing via omp Last update from 192.168.20.1 00:15:16 ago Routing Descriptor Blocks: * 192.168.20.1 (default), from 192.168.20.1, 00:15:16 ago opaque_ptr 0x7F43A2658A00 Route metric is 0, traffic share count is 1 AS Hops 0 MPLS label: 97 MPLS Flags: MPLS Required DC0-SDWAN-LAB01#show bgp vrf 40 vpnv4 unicast 220.127.116.11 BGP routing table entry for 1:40:18.104.22.168/32, version 23 Paths: (1 available, best #1, table 40) Not advertised to any peer Refresh Epoch 6 Local, imported path from 192.168.255.61:3006:22.214.171.124/32 (global) 192.168.20.1 (via default) from 192.168.20.1 (192.168.255.61) Origin IGP, metric 0, localpref 100, valid, internal, best Extended Community: RT:65106:3006 RT:65106:10003006 mpls labels in/out nolabel/97 rx pathid: 0, tx pathid: 0x0 Updated on Feb 15 2022 23:32:45 BST
On the ASR router as expected the inner label is 97 which is what is advertised by BGP to the SD-WAN router.
DC0-ASR-LAB01#show bgp vrf GRY vpnv4 unicast 126.96.36.199 BGP routing table entry for 192.168.255.61:3006:188.8.131.52/32, version 5077 Paths: (1 available, best #1, table GRY) Multipath: eBGP Advertised to update-groups: 823 824 Refresh Epoch 1 Local 0.0.0.0 (via vrf GRY) from 0.0.0.0 (192.168.255.61) Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best Extended Community: RT:65106:3006 RT:65106:10003006 Local vxlan vtep: vrf:GRY, vni:10003006 local router mac:B838.61F5.F1BF encap:8 vtep-ip:192.168.255.65 bdi:BDI3006 mpls labels in/out 97/nolabel(GRY) rx pathid: 0, tx pathid: 0x0 DC0-ASR-LAB01#show mpls forwarding-table vrf GRY Local Outgoing Prefix Bytes Label Outgoing Next Hop Label Label or Tunnel Id Switched interface 97 Pop Label 184.108.40.206/32[V] 0 aggregate/GRY
The outer label learnt via BGP from the SD-WAN router is 100000.
DC0-ASR-LAB01#show ip route vrf GRY 220.127.116.11 Routing Table: GRY Routing entry for 18.104.22.168/32 Known via "bgp 65106", distance 200, metric 0, type internal Last update from 192.168.20.2 00:07:52 ago Routing Descriptor Blocks: * 192.168.20.2 (default), from 192.168.20.2, 00:07:52 ago Route metric is 0, traffic share count is 1 AS Hops 0 MPLS label: 100000 MPLS Flags: MPLS Required, NSF DC0-ASR-LAB01#show bgp vrf GRY vpnv4 unicast 22.214.171.124 BGP routing table entry for 192.168.255.61:3006:126.96.36.199/32, version 5079 Paths: (1 available, best #1, table GRY) Multipath: eBGP Advertised to update-groups: 823 Refresh Epoch 1 Local, (Received from a RR-client), imported path from 1:40:188.8.131.52/32 (global) 192.168.20.2 (via default) from 192.168.20.2 (192.168.20.2) Origin IGP, metric 0, localpref 100, valid, internal, best Extended Community: RT:65106:40 Local vxlan vtep: vrf:GRY, vni:10003006 local router mac:B838.61F5.F1BF encap:8 vtep-ip:192.168.255.65 bdi:BDI3006 mpls labels in/out nolabel/100000 rx pathid: 0, tx pathid: 0x0
In terms of label capability the SD-WAN device doesn’t have the neighbor x.x.x.x send-label command which is why it shows only received. I was expecting that it would have this command, but looks to work without it.
DC0-SDWAN-LAB01#show ip bgp neighbors | in Label ipv4 MPLS Label capability: received DC0-ASR-LAB01#show ip bgp neighbors | in Label ipv4 MPLS Label capability: advertised Sending Prefix & Label