A recent project I was working on involved the need to join a new office to our existing Data Centres and OSPF core using a Gig circuit over the Internet. To flesh out this idea and test its viability I thought I would try and solve an ESX capacity problem I have at home by moving vCentre into the cloud.
Table Of Contents
When ESX 6.5 came out I looked at upgrading but could never get vCentre to run stably on anything less than the 8GB of RAM. I use a NUC for my lab ESX host which is limited to a max of 32GB RAM, therefore wasting a quarter of that for features I occasionally use didn’t make sense. I can get way with only 3GB for vCentre on ESX5.5. Unfortunately win10 and ESX5.5 don’t seem to work well together, therefore the need to upgrade to 6.5 has become more urgent.
The initial setup will use Server2012 (vcentre) and CSR1000v (GRE & IPsec) in Azure, whilst at home an ASA5505 (IPsec) and CSR1000v (GRE). I could have terminated the IPsec tunnel on the CSR at both ends, however at work we have Nexus7K on one end that doesn’t support IPsec so am using the ASA to replicate our production environment.
I am using Azure over AWS simply because I have a Visual Studio Enterprise subscription which I get free credits with, I am sure you could do the exact same thing in AWS. In the future once I am finished with testing I will remove the CSRs and use Azure VPN as the CSRs are quite pricey and trial license is only 3 months at a time.
There are 3 core elements to this solution:
- IPsec tunnel that will allow only the GRE traffic.
- GRE tunnel which will allow all traffic between home and Azure.
- OSPF peering over the GRE tunnel.
1. License CSRs
By default when you deploy the CSRs they have an AX license, meaning you can use all the features but are limited to only 100kbps of throughput.
show platform hardware throughput level
Check current throughputshow version | in license To see if it has a valid license
You can obtain a free 60 day trial license for cisco, however is limited to 6 before you have to beg Cisco support for more.
The 3 main types of license are:
IPBase: Basic Networking Routing (Routing, HSRP, NAT, ACL, VRF, GRE)
Security: IPBase package + Security features (IP Security VPN, Firewall, MPLS, Multicast, QoS)
AX: IPBase package + Security features + Advanced Networking features (AppNav, AVC, OTV and LISP)\
To get license first need get the UID from the CSR, this is what the license is bound to like a serial number with physical devices).
show license udi
Go to Cisco licensing Get Licenses » Demo and Evaluation » routers & switches » cisco cloud services router 1000v, choose 2.5 Gbps Full Featured (AX) 60 day evaluation license and enter the UID.
Upload the license to the CSR and install it, you shouldn’t need a reboot unless your changing the license level.
bootflash:lic_name.xmllicense boot level ax Change the boot level if not axshow license detailshow version | in licenseshow platform hardware throughput level Check license throughput
2. HME-ASA1 to HME-CSR1 connectivity
I am running OSPF area0 between these devices and in the core of my network. Loopback0 is advertised into this area as it will be used for the GRE tunnel source/destination.
interface Vlan10 nameif mgmt security-level 100 ip address 10.10.10.1 255.255.255.0 ! interface Vlan101 nameif gre-transit security-level 100 ip address 126.96.36.199 255.255.255.240 ! route-map CONN->OSPF permit 10 match interface mgmt ! router ospf 1 router-id 188.8.131.52 network 184.108.40.206 255.255.255.240 area 0 redistribute connected subnets route-map CONN->OSPF
HME-ASA1# show ospf int brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C gre-transit 1 0 220.127.116.11/255.255.255.240 10 DR 1HME-ASA1# show ospf neighbor Neighbor ID Pri State Dead Time Address Interface 192.168.201.1 1 FULL/BDR 0:00:34 18.104.22.168 gre-transit
router ospf 1 router-id 192.168.201.1 ! interface GigabitEthernet1 ip address 22.214.171.124 255.255.255.240 ip ospf 1 area 0 ! interface Loopback1 ip address 192.168.201.1 255.255.255.255 ip ospf 1 area 0
HME-CSR1# show ip ospf int bri
Interface PID Area IP Address/Mask Cost State Nbrs F/C Lo1 1 0 192.168.201.1/32 1 LOOP 0/0 Gi1 1 0 126.96.36.199/28 1 BDR 1/1HME-CSR1# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 188.8.131.52 1 FULL/DR 00:00:36 184.108.40.206 GigabitEthernet1
3. ASA IPsec configuration
The IPsec VPN interesting traffic only needs to carry the CSR loopback interfaces that will be used as the source and destination by the GRE tunnel.
object-group network HME-to-AZE-LOCAL network-object host 192.168.201.1 object-group network HME-to-AZE-REMOTE network-object host 220.127.116.11 ! access-list VPN-HME-to-AZE extended permit ip object-group HME-to-AZE-LOCAL object-group HME-to-AZE-REMOTE
If sysopt connection permit-vpn is disabled a permit rule is also needed in the outside ACL.
access-list outside extended permit ip object-group HME-to-AZE-REMOTE object-group HME-to-AZE-LOCAL
Will need to noNAT the traffic if it is matched by another NAT such as PAT. Easy way to check is with packet-tracer.
packet-tracer input gre-transit udp 192.168.201.1 500 18.104.22.168 500
Is best to put this rule near the top of the NAT rules as you don’t want this traffic to be accidentally matched by another NAT in the future.
nat (gre-trasnit,outside) source static HME-to-AZE-LOCAL HME-to-AZE-LOCAL destination static HME-to-AZE-REMOTE HME-to-AZE-REMOTE
Phase1 and Phase2 authentication and encryption algorithms
crypto ikev1 policy 10 authentication pre-share encryption aes-256 hash sha group 5 lifetime 86400 crypto ikev1 enable outside ! crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac crypto map outside_map 10 match address VPN-HME-to-AZE crypto map outside_map 10 set pfs group5 crypto map outside_map 10 set peer <azure-public-ip> crypto map outside_map 10 set ikev1 transform-set ESP-AES-256-SHA crypto map outside_map interface outside ! tunnel-group <azure-public-ip> type ipsec-l2l tunnel-group <azure-public-ip> ipsec-attributes ikev1 pre-shared-key <myvpnisnotsecure>
4. Azure AZE-CSR1 base and IPsec configuration
The Internet facing interface of the CSR will be in its own VRF to ensure there is isolation between the Internet and the private network. This also allows a default route to be advertised over OSPF making all traffic in this network segment at the remote site uses the GRE tunnel whilst ensuring it doesn’t affect the GRE control traffic.
As the loopback is used to source the GRE tunnel that traverses the Internet it will also have to in this VRF.
Normally the public facing interface on the CSR would be statically assigned, in Azure this has to be DHCP.
vrf definition INET rd 1:0 address-family ipv4 ! interface GigabitEthernet1 vrf forwarding INET ip address dhcp ! interface Loopback1 vrf forwarding INET ip address 22.214.171.124 255.255.255.255 ! ip route vrf INET 0.0.0.0 0.0.0.0 126.96.36.199
object-group ip access-list extended VPN-HME-to-AZE permit ip host 188.8.131.52 host 192.168.201.1
Phase1 and Phase2 authentication and encryption algorithms
crypto isakmp policy 10 encr aes 256 authentication pre-share group 5 ! crypto isakmp keepalive 10 2 crypto ipsec transform-set ESP-AES-256-SHA esp-aes 256 esp-sha-hmac crypto map INET 1 ipsec-isakmp set peer <home-public-ip> set transform-set ESP-AES-256-SHA set pfs group5 match address VPN-HME-to-AZE ! interface GigabitEthernet1 crypto map INET
As a VRF is being used for the transport a keyring is required for the pre-shared key.
crypto keyring FG-to-DC1 vrf INET pre-shared-key address <home-public-ip> 255.255.255.255 key <myvpnisnotsecure>
The CSR required the DPD keepalive command where as the ASA had this by default. The least this can be set to is an interval of 10 seconds and retry twice. So in theory if the VPN had issues and was stuck in a state it would take a max of 29 seconds for this to cleared.
The tunnel can now be brought up by pinging across it, should see encapsulated and decapsulated packets
AZE-CSR1# ping vrf INET 192.168.201.1 source loop1
Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.201.1, timeout is 2 seconds: Packet sent with a source address of 184.108.40.206 !!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 17/17/19 msAZE-CSR1# show crypto session detail Crypto session current status Code: C - IKE Configuration mode, D - Dead Peer Detection K - Keepalives, N - NAT-traversal, T - cTCP encapsulation X - IKE Extended Authentication, F - IKE Fragmentation R - IKE Auto Reconnect Interface: GigabitEthernet1 Uptime: 00:32:07 Session status: UP-ACTIVE Peer: 220.127.116.11 port 4500 fvrf: INET ivrf: INET Phase1_id: 18.104.22.168 Desc: (none) Session ID: 0 IKEv1 SA: local 22.214.171.124/4500 remote 126.96.36.199/4500 Active Capabilities:DKN connid:1049 lifetime:23:27:51 IPSEC FLOW: permit ip host 188.8.131.52 host 192.168.201.1 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 573576 drop 0 life (KB/Sec) 4607909/1672 Outbound: #pkts enc'ed 511359 drop 0 life (KB/Sec) 4607946/1672HME-ASA1# show vpn-sessiondb l2l Session Type: LAN-to-LAN Connection : 184.108.40.206 Index : 33 IP Addr : 220.127.116.11 Protocol : IKEv1 IPsecOverNatT Encryption : IKEv1: (1)AES256 IPsecOverNatT: (1)AES256 Hashing : IKEv1: (1)SHA1 IPsecOverNatT: (1)SHA1 Bytes Tx : 61476 Bytes Rx : 61600 Login Time : 10:31:53 GMT Sun Nov 26 2017 Duration : 0h:36m:09s
5. GRE Tunnels
Now the IPsec tunnel is up just need to build the GRE tunnel on the Home CSR and the Azure CSR using the loopbacks configured earlier and run OPSF area 10 over this.
To account for the GRE (24 bytes) and ESP (upto 75 bytes dependant on algorithms used) overhead it is recommended to lower the MTU to 1400 bytes and TCP Maximum Segment Size (MSS) to 1360 bytes (IP header 20 Bytes, TCP/UDP header 20 Bytes). This will help to ensure that the end devices do the fragmentation rather than the network devices having to do it for every packet.
The Crypto traffic of the tunnel is to be in the INET VRF yet the tunnel itself is in the default global VRF. This is accomplished with the command tunnel vrf INET that specifies the tunnel crypto traffic will be through this VRF.
GRE keepalives are used to dynamically bring down the tunnel interface if the remote end is down. These timers are set to match the IPsec DPD keepalives as the GRE tunnel relies on the IPsec tunnel as its mode of transport.
router ospf 1 router-id 192.168.201.1 passive-interface GigabitEthernet2 ! interface GigabitEthernet2 ip address dhcp ip ospf 1 area 10 ! interface Tunnel10 ip address 18.104.22.168 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf hello-interval 3 ip ospf 1 area 10 keepalive 10 2 tunnel source Loopback1 tunnel destination 192.168.201.1 tunnel vrf INET
interface Tunnel10 ip address 22.214.171.124 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf hello-interval 5 ip ospf 1 area 10 keepalive 10 2 tunnel source Loopback1 tunnel destination 126.96.36.199
Home CSR verification
HME-CSR1# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 188.8.131.52 1 FULL/DR 00:00:39 184.108.40.206 GigabitEthernet1 220.127.116.11 0 FULL/ - 00:00:17 18.104.22.168 Tunnel10HME-CSR1# show ip ospf interface brief Interface PID Area IP Address/Mask Cost State Nbrs F/C Lo1 1 0 192.168.201.1/32 1 LOOP 0/0 Gi1 1 0 22.214.171.124/28 1 BDR 1/1 Tu10 1 10 126.96.36.199/30 1000 P2P 1/1HME-CSR1# show ip route ospf Gateway of last resort is 188.8.131.52 to network 0.0.0.0 O*E1 0.0.0.0/0 110/12via 184.108.40.206, 04:48:39, GigabitEthernet1 10.0.0.0/24 is subnetted, 1 subnets O E1 10.10.10.0 110/12via 220.127.116.11, 04:48:39, GigabitEthernet1 18.104.22.168/24 is subnetted, 1 subnets O 22.214.171.124 110/1001via 126.96.36.199, 04:08:31, Tunnel10 188.8.131.52/16 is variably subnetted, 5 subnets, 3 masks O 184.108.40.206/28 110/11via 220.127.116.11, 04:48:39, GigabitEthernet1
Azure CSR verification
AZE-CSR1# show ip ospf int brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C Gi2 1 10 18.104.22.168/24 1 DR 0/0 Tu10 1 10 22.214.171.124/30 1000 P2P 1/1AZE-CSR1# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 192.168.201.1 0 FULL/ - 00:00:18 126.96.36.199 Tunnel10AZE-CSR1#show ip route ospf Gateway of last resort is 188.8.131.52 to network 0.0.0.0 O*E1 0.0.0.0/0 110/1012via 184.108.40.206, 04:11:20, Tunnel10 10.0.0.0/24 is subnetted, 5 subnets O E1 10.10.10.0 110/1012via 220.127.116.11, 04:11:20, Tunnel10 18.104.22.168/16 is variably subnetted, 3 subnets, 3 masks O IA 22.214.171.124/28 110/1001via 126.96.36.199, 04:11:20, Tunnel10 192.168.201.0/32 is subnetted, 1 subnets O IA 192.168.201.1 110/1001via 188.8.131.52, 04:11:20, Tunnel10
6. Quality of Service and Zone-Based Firewall
Enabling qos pre-classify under the tunnel interface would ensure that the TOS byte is passed onto the ESP packets, however it can only be used if IPsec and GRE tunnels terminate on the same device.
As QoS only kicks in if the interface is congested an alternative method is to use a child policy that allows you to cause congestion at less that the interfaces physical speed. The below example shapes at 100Mbps and then applies the QoS as per the child policy.
policy-map PM-CHILD class CONTROL bandwidth percent 5 class class-default ! policy-map PM-PARENT class class-default shape average percent 100 service-policy PM-CHILD ! int tun 10 bandwidth qos-reference 1000000 service-policy output PM-PARENT
As the Azure CSR is facing the Internet it needs locking down in some manner. Could use Azure furewall rules or the CSR ZBFW to ensure that the only traffic the CSR will accept is IPsec and GRE from the remote peer. Although it is a bit clunky to configure, as it is unlikely to change much the CSR ZBFW is a good place to do the packet filtering so that all the configuration is in the one place.
ip access-list extended VRF_OUTSIDE->SELF permit icmp any any echo permit icmp any any echo-reply permit icmp any any port-unreachable permit icmp any any time-exceeded remark IPSEC TRAFFIC permit esp host <home-public-ip> host <azure-public-ip> permit udp host <home-public-ip> host <azure-public-ip> eq isakmp permit udp host <home-public-ip> host <azure-public-ip> eq 4500 remark GRE TRAFFIC permit gre host 192.168.201.1 host 184.108.40.206 ! class-map type inspect match-all CM-VRF_OUTSIDE->SELF match access-group name VRF_OUTSIDE->SELF ! policy-map type inspect PM-VRF_OUTSIDE->SELF class type inspect CM-VRF_OUTSIDE->SELF inspect class class-default drop ! zone security VRF_OUTSIDE ! zone-pair security ZP_VRF_OUTSIDE->SELF source VRF_OUTSIDE destination self service-policy type inspect PM-VRF_OUTSIDE->SELF ! ip access-list extended SELF->VRF_OUTSIDE permit icmp any any permit udp any any range 33434 33464 remark IPSEC TRAFFIC permit esp host <azure-public-ip> host <home-public-ip> permit udp host <azure-public-ip> host <home-public-ip> eq isakmp permit udp host <azure-public-ip> host <home-public-ip> eq 4500 remark GRE TRAFFIC permit gre host 220.127.116.11 host 192.168.201.1 ! class-map type inspect match-all CM-SELF->VRF_OUTSIDE match access-group name SELF->VRF_OUTSIDE ! policy-map type inspect PM-SELF->VRF_OUTSIDE class type inspect CM-SELF->VRF_OUTSIDE inspect class class-default drop ! zone-pair security ZP_SELF->VRF_OUTSIDE source self destination VRF_OUTSIDE service-policy type inspect PM-SELF->VRF_OUTSIDE ! int gi 1 zone-member security VRF_OUTSIDE