A recent project I was working on involved the need to join a new office to our existing Data Centres and OSPF core using a Gig circuit over the Internet. To flesh out this idea and test its viability I thought I would try and solve an ESX capacity problem I have at home by moving vCentre into the cloud.
Table Of Contents
When ESX 6.5 came out I looked at upgrading but could never get vCentre to run stably on anything less than the 8GB of RAM. I use a NUC for my lab ESX host which is limited to a max of 32GB RAM, therefore wasting a quarter of that for features I occasionally use didn’t make sense. I can get way with only 3GB for vCentre on ESX5.5. Unfortunately win10 and ESX5.5 don’t seem to work well together, therefore the need to upgrade to 6.5 has become more urgent.
The initial setup will use Server2012 (vcentre) and CSR1000v (GRE & IPsec) in Azure, whilst at home an ASA5505 (IPsec) and CSR1000v (GRE). I could have terminated the IPsec tunnel on the CSR at both ends, however at work we have Nexus7K on one end that doesn’t support IPsec so am using the ASA to replicate our production environment.
I am using Azure over AWS simply because I have a Visual Studio Enterprise subscription which I get free credits with, I am sure you could do the exact same thing in AWS. In the future once I am finished with testing I will remove the CSRs and use Azure VPN as the CSRs are quite pricey and trial license is only 3 months at a time.
There are 3 core elements to this solution:
- IPsec tunnel that will allow only the GRE traffic.
- GRE tunnel which will allow all traffic between home and Azure.
- OSPF peering over the GRE tunnel.
1. License CSRs
By default when you deploy the CSRs they have an AX license, meaning you can use all the features but are limited to only 100kbps of throughput.
show platform hardware throughput levelCheck current throughput show version | in licenseTo see if it has a valid license
You can obtain a free 60 day trial license for cisco, however is limited to 6 before you have to beg Cisco support for more.
The 3 main types of license are:
IPBase: Basic Networking Routing (Routing, HSRP, NAT, ACL, VRF, GRE)
Security: IPBase package + Security features (IP Security VPN, Firewall, MPLS, Multicast, QoS)
AX: IPBase package + Security features + Advanced Networking features (AppNav, AVC, OTV and LISP)\
To get license first need get the UID from the CSR, this is what the license is bound to like a serial number with physical devices).
show license udi
Go to Cisco licensing Get Licenses » Demo and Evaluation » routers & switches » cisco cloud services router 1000v, choose 2.5 Gbps Full Featured (AX) 60 day evaluation license and enter the UID.
Upload the license to the CSR and install it, you shouldn’t need a reboot unless your changing the license level.
license installbootflash:lic_name.xml license boot levelax Change the boot level if not ax show license detailshow version | in licenseshow platform hardware throughput levelCheck license throughput
2. HME-ASA1 to HME-CSR1 connectivity
I am running OSPF area0 between these devices and in the core of my network. Loopback0 is advertised into this area as it will be used for the GRE tunnel source/destination.
HME-ASA1
interface Vlan10
nameif mgmt
security-level 100
ip address 10.10.10.1 255.255.255.0
!
interface Vlan101
nameif gre-transit
security-level 100
ip address 172.168.255.17 255.255.255.240
!
route-map CONN->OSPF permit 10
match interface mgmt
!
router ospf 1
router-id 172.168.255.1
network 172.168.255.16 255.255.255.240 area 0
redistribute connected subnets route-map CONN->OSPF
HME-ASA1# show ospf int briefInterface PID Area IP Address/Mask Cost State Nbrs F/C gre-transit 1 0 172.168.255.17/255.255.255.240 10 DR 1 HME-ASA1# show ospf neighborNeighbor ID Pri State Dead Time Address Interface 192.168.201.1 1 FULL/BDR 0:00:34 172.168.255.18 gre-transit
HME-CSR1
router ospf 1
router-id 192.168.201.1
!
interface GigabitEthernet1
ip address 172.168.255.18 255.255.255.240
ip ospf 1 area 0
!
interface Loopback1
ip address 192.168.201.1 255.255.255.255
ip ospf 1 area 0
HME-CSR1# show ip ospf int briInterface PID Area IP Address/Mask Cost State Nbrs F/C Lo1 1 0 192.168.201.1/32 1 LOOP 0/0 Gi1 1 0 172.168.255.18/28 1 BDR 1/1 HME-CSR1# show ip ospf neighborNeighbor ID Pri State Dead Time Address Interface 172.168.255.1 1 FULL/DR 00:00:36 172.168.255.17 GigabitEthernet1
3. ASA IPsec configuration
The IPsec VPN interesting traffic only needs to carry the CSR loopback interfaces that will be used as the source and destination by the GRE tunnel.
Interesting traffic
object-group network HME-to-AZE-LOCAL
network-object host 192.168.201.1
object-group network HME-to-AZE-REMOTE
network-object host 172.168.201.1
!
access-list VPN-HME-to-AZE extended permit ip object-group HME-to-AZE-LOCAL object-group HME-to-AZE-REMOTE
If sysopt connection permit-vpn is disabled a permit rule is also needed in the outside ACL.
access-list outside extended permit ip object-group HME-to-AZE-REMOTE object-group HME-to-AZE-LOCAL
Will need to noNAT the traffic if it is matched by another NAT such as PAT. Easy way to check is with packet-tracer.
packet-tracer input gre-transit udp 192.168.201.1 500 172.168.201.1 500
Is best to put this rule near the top of the NAT rules as you don’t want this traffic to be accidentally matched by another NAT in the future.
nat (gre-trasnit,outside) source static HME-to-AZE-LOCAL HME-to-AZE-LOCAL destination static HME-to-AZE-REMOTE HME-to-AZE-REMOTE
Phase1 and Phase2 authentication and encryption algorithms
crypto ikev1 policy 10
authentication pre-share
encryption aes-256
hash sha
group 5
lifetime 86400
crypto ikev1 enable outside
!
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto map outside_map 10 match address VPN-HME-to-AZE
crypto map outside_map 10 set pfs group5
crypto map outside_map 10 set peer <azure-public-ip>
crypto map outside_map 10 set ikev1 transform-set ESP-AES-256-SHA
crypto map outside_map interface outside
!
tunnel-group <azure-public-ip> type ipsec-l2l
tunnel-group <azure-public-ip> ipsec-attributes
ikev1 pre-shared-key <myvpnisnotsecure>
4. Azure AZE-CSR1 base and IPsec configuration
The Internet facing interface of the CSR will be in its own VRF to ensure there is isolation between the Internet and the private network. This also allows a default route to be advertised over OSPF making all traffic in this network segment at the remote site uses the GRE tunnel whilst ensuring it doesn’t affect the GRE control traffic.
As the loopback is used to source the GRE tunnel that traverses the Internet it will also have to in this VRF.
Normally the public facing interface on the CSR would be statically assigned, in Azure this has to be DHCP.
vrf definition INET
rd 1:0
address-family ipv4
!
interface GigabitEthernet1
vrf forwarding INET
ip address dhcp
!
interface Loopback1
vrf forwarding INET
ip address 172.168.201.1 255.255.255.255
!
ip route vrf INET 0.0.0.0 0.0.0.0 172.60.99.1
Interesting traffic
object-group ip access-list extended VPN-HME-to-AZE
permit ip host 172.168.201.1 host 192.168.201.1
Phase1 and Phase2 authentication and encryption algorithms
crypto isakmp policy 10
encr aes 256
authentication pre-share
group 5
!
crypto isakmp keepalive 10 2
crypto ipsec transform-set ESP-AES-256-SHA esp-aes 256 esp-sha-hmac
crypto map INET 1 ipsec-isakmp
set peer <home-public-ip>
set transform-set ESP-AES-256-SHA
set pfs group5
match address VPN-HME-to-AZE
!
interface GigabitEthernet1
crypto map INET
As a VRF is being used for the transport a keyring is required for the pre-shared key.
crypto keyring FG-to-DC1 vrf INET
pre-shared-key address <home-public-ip> 255.255.255.255 key <myvpnisnotsecure>
The CSR required the DPD keepalive command where as the ASA had this by default. The least this can be set to is an interval of 10 seconds and retry twice. So in theory if the VPN had issues and was stuck in a state it would take a max of 29 seconds for this to cleared.
The tunnel can now be brought up by pinging across it, should see encapsulated and decapsulated packets
AZE-CSR1# ping vrf INET 192.168.201.1 source loop1Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.201.1, timeout is 2 seconds: Packet sent with a source address of 172.168.201.1 !!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 17/17/19 ms AZE-CSR1# show crypto session detailCrypto session current status Code: C - IKE Configuration mode, D - Dead Peer Detection K - Keepalives, N - NAT-traversal, T - cTCP encapsulation X - IKE Extended Authentication, F - IKE Fragmentation R - IKE Auto Reconnect Interface: GigabitEthernet1 Uptime: 00:32:07 Session status: UP-ACTIVE Peer: 87.115.228.25 port 4500 fvrf: INET ivrf: INET Phase1_id: 172.168.255.1 Desc: (none) Session ID: 0 IKEv1 SA: local 172.60.99.4/4500 remote 87.115.228.25/4500 Active Capabilities:DKN connid:1049 lifetime:23:27:51 IPSEC FLOW: permit ip host 172.168.201.1 host 192.168.201.1 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 573576 drop 0 life (KB/Sec) 4607909/1672 Outbound: #pkts enc'ed 511359 drop 0 life (KB/Sec) 4607946/1672 HME-ASA1# show vpn-sessiondb l2lSession Type: LAN-to-LAN Connection : 51.144.156.155 Index : 33 IP Addr : 51.144.156.155 Protocol : IKEv1 IPsecOverNatT Encryption : IKEv1: (1)AES256 IPsecOverNatT: (1)AES256 Hashing : IKEv1: (1)SHA1 IPsecOverNatT: (1)SHA1 Bytes Tx : 61476 Bytes Rx : 61600 Login Time : 10:31:53 GMT Sun Nov 26 2017 Duration : 0h:36m:09s
5. GRE Tunnels
Now the IPsec tunnel is up just need to build the GRE tunnel on the Home CSR and the Azure CSR using the loopbacks configured earlier and run OPSF area 10 over this.
To account for the GRE (24 bytes) and ESP (upto 75 bytes dependant on algorithms used) overhead it is recommended to lower the MTU to 1400 bytes and TCP Maximum Segment Size (MSS) to 1360 bytes (IP header 20 Bytes, TCP/UDP header 20 Bytes). This will help to ensure that the end devices do the fragmentation rather than the network devices having to do it for every packet.
The Crypto traffic of the tunnel is to be in the INET VRF yet the tunnel itself is in the default global VRF. This is accomplished with the command tunnel vrf INET that specifies the tunnel crypto traffic will be through this VRF.
GRE keepalives are used to dynamically bring down the tunnel interface if the remote end is down. These timers are set to match the IPsec DPD keepalives as the GRE tunnel relies on the IPsec tunnel as its mode of transport.
AZE-CSR1
router ospf 1
router-id 192.168.201.1
passive-interface GigabitEthernet2
!
interface GigabitEthernet2
ip address dhcp
ip ospf 1 area 10
!
interface Tunnel10
ip address 172.168.220.2 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
ip ospf hello-interval 3
ip ospf 1 area 10
keepalive 10 2
tunnel source Loopback1
tunnel destination 192.168.201.1
tunnel vrf INET
HME-CSR1
interface Tunnel10
ip address 172.168.220.1 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
ip ospf hello-interval 5
ip ospf 1 area 10
keepalive 10 2
tunnel source Loopback1
tunnel destination 172.168.201.1
Home CSR verification
HME-CSR1# show ip ospf neighborNeighbor ID Pri State Dead Time Address Interface 172.168.255.1 1 FULL/DR 00:00:39 172.168.255.17 GigabitEthernet1 172.168.201.1 0 FULL/ - 00:00:17 172.168.220.2 Tunnel10 HME-CSR1# show ip ospf interface briefInterface PID Area IP Address/Mask Cost State Nbrs F/C Lo1 1 0 192.168.201.1/32 1 LOOP 0/0 Gi1 1 0 172.168.255.18/28 1 BDR 1/1 Tu10 1 10 172.168.220.1/30 1000 P2P 1/1 HME-CSR1# show ip route ospfGateway of last resort is 172.168.255.17 to network 0.0.0.0 O*E1 0.0.0.0/0 110/12 via 172.168.255.17, 04:48:39, GigabitEthernet110.0.0.0/24 is subnetted, 1 subnets O E1 10.10.10.0 110/12 via 172.168.255.17, 04:48:39, GigabitEthernet1172.60.0.0/24 is subnetted, 1 subnets O 172.60.1.0 110/1001 via 172.168.220.2, 04:08:31, Tunnel10172.168.0.0/16 is variably subnetted, 5 subnets, 3 masks O 172.168.255.0/28 110/11 via 172.168.255.17, 04:48:39, GigabitEthernet1
Azure CSR verification
AZE-CSR1# show ip ospf int briefInterface PID Area IP Address/Mask Cost State Nbrs F/C Gi2 1 10 172.60.1.4/24 1 DR 0/0 Tu10 1 10 172.168.220.2/30 1000 P2P 1/1 AZE-CSR1# show ip ospf neighborNeighbor ID Pri State Dead Time Address Interface 192.168.201.1 0 FULL/ - 00:00:18 172.168.220.1 Tunnel10 AZE-CSR1#show ip route ospfGateway of last resort is 172.168.220.1 to network 0.0.0.0 O*E1 0.0.0.0/0 110/1012 via 172.168.220.1, 04:11:20, Tunnel1010.0.0.0/24 is subnetted, 5 subnets O E1 10.10.10.0 110/1012 via 172.168.220.1, 04:11:20, Tunnel10172.168.0.0/16 is variably subnetted, 3 subnets, 3 masks O IA 172.168.255.16/28 110/1001 via 172.168.220.1, 04:11:20, Tunnel10192.168.201.0/32 is subnetted, 1 subnets O IA 192.168.201.1 110/1001 via 172.168.220.1, 04:11:20, Tunnel10
6. Quality of Service and Zone-Based Firewall
Enabling qos pre-classify under the tunnel interface would ensure that the TOS byte is passed onto the ESP packets, however it can only be used if IPsec and GRE tunnels terminate on the same device.
As QoS only kicks in if the interface is congested an alternative method is to use a child policy that allows you to cause congestion at less that the interfaces physical speed. The below example shapes at 100Mbps and then applies the QoS as per the child policy.
policy-map PM-CHILD
class CONTROL
bandwidth percent 5
class class-default
!
policy-map PM-PARENT
class class-default
shape average percent 100
service-policy PM-CHILD
!
int tun 10
bandwidth qos-reference 1000000
service-policy output PM-PARENT
As the Azure CSR is facing the Internet it needs locking down in some manner. Could use Azure furewall rules or the CSR ZBFW to ensure that the only traffic the CSR will accept is IPsec and GRE from the remote peer. Although it is a bit clunky to configure, as it is unlikely to change much the CSR ZBFW is a good place to do the packet filtering so that all the configuration is in the one place.
ip access-list extended VRF_OUTSIDE->SELF
permit icmp any any echo
permit icmp any any echo-reply
permit icmp any any port-unreachable
permit icmp any any time-exceeded
remark IPSEC TRAFFIC
permit esp host <home-public-ip> host <azure-public-ip>
permit udp host <home-public-ip> host <azure-public-ip> eq isakmp
permit udp host <home-public-ip> host <azure-public-ip> eq 4500
remark GRE TRAFFIC
permit gre host 192.168.201.1 host 172.168.201.1
!
class-map type inspect match-all CM-VRF_OUTSIDE->SELF
match access-group name VRF_OUTSIDE->SELF
!
policy-map type inspect PM-VRF_OUTSIDE->SELF
class type inspect CM-VRF_OUTSIDE->SELF
inspect
class class-default
drop
!
zone security VRF_OUTSIDE
!
zone-pair security ZP_VRF_OUTSIDE->SELF source VRF_OUTSIDE destination self
service-policy type inspect PM-VRF_OUTSIDE->SELF
!
ip access-list extended SELF->VRF_OUTSIDE
permit icmp any any
permit udp any any range 33434 33464
remark IPSEC TRAFFIC
permit esp host <azure-public-ip> host <home-public-ip>
permit udp host <azure-public-ip> host <home-public-ip> eq isakmp
permit udp host <azure-public-ip> host <home-public-ip> eq 4500
remark GRE TRAFFIC
permit gre host 172.168.201.1 host 192.168.201.1
!
class-map type inspect match-all CM-SELF->VRF_OUTSIDE
match access-group name SELF->VRF_OUTSIDE
!
policy-map type inspect PM-SELF->VRF_OUTSIDE
class type inspect CM-SELF->VRF_OUTSIDE
inspect
class class-default
drop
!
zone-pair security ZP_SELF->VRF_OUTSIDE source self destination VRF_OUTSIDE
service-policy type inspect PM-SELF->VRF_OUTSIDE
!
int gi 1
zone-member security VRF_OUTSIDE