vCentre in Azure

vcentre in azure over ipsec and gre

26 November 2017   11 min read

A recent project I was working on involved the need to join a new office to our existing Data Centres and OSPF core using a Gig circuit over the Internet. To flesh out this idea and test its viability I thought I would try and solve an ESX capacity problem I have at home by moving vCentre into the cloud.



When ESX 6.5 came out I looked at upgrading but could never get vCentre to run stably on anything less than the 8GB of RAM. I use a NUC for my lab ESX host which is limited to a max of 32GB RAM, therefore wasting a quarter of that for features I occasionally use didn’t make sense. I can get way with only 3GB for vCentre on ESX5.5. Unfortunately win10 and ESX5.5 don’t seem to work well together, therefore the need to upgrade to 6.5 has become more urgent.

The initial setup will use Server2012 (vcentre) and CSR1000v (GRE & IPsec) in Azure, whilst at home an ASA5505 (IPsec) and CSR1000v (GRE). I could have terminated the IPsec tunnel on the CSR at both ends, however at work we have Nexus7K on one end that doesn’t support IPsec so am using the ASA to replicate our production environment.

I am using Azure over AWS simply because I have a Visual Studio Enterprise subscription which I get free credits with, I am sure you could do the exact same thing in AWS. In the future once I am finished with testing I will remove the CSRs and use Azure VPN as the CSRs are quite pricey and trial license is only 3 months at a time.

There are 3 core elements to this solution:

  • IPsec tunnel that will allow only the GRE traffic.
  • GRE tunnel which will allow all traffic between home and Azure.
  • OSPF peering over the GRE tunnel.

1. License CSRs

By default when you deploy the CSRs they have an AX license, meaning you can use all the features but are limited to only 100kbps of throughput.

show platform hardware throughput level                                    Check current throughput
show version | in license         To see if it has a valid license

You can obtain a free 60 day trial license for cisco, however is limited to 6 before you have to beg Cisco support for more.

The 3 main types of license are:
IPBase: Basic Networking Routing (Routing, HSRP, NAT, ACL, VRF, GRE)
Security: IPBase package + Security features (IP Security VPN, Firewall, MPLS, Multicast, QoS)
AX: IPBase package + Security features + Advanced Networking features (AppNav, AVC, OTV and LISP)\

To get license first need get the UID from the CSR, this is what the license is bound to like a serial number with physical devices).

show license udi

Go to Cisco licensing Get Licenses » Demo and Evaluation » routers & switches » cisco cloud services router 1000v, choose 2.5 Gbps Full Featured (AX) 60 day evaluation license and enter the UID.

Upload the license to the CSR and install it, you shouldn’t need a reboot unless your changing the license level.

license install bootflash:lic_name.xml
license boot level ax         Change the boot level if not ax

show license detail
show version | in license
show platform hardware throughput level Check license throughput

2. HME-ASA1 to HME-CSR1 connectivity

I am running OSPF area0 between these devices and in the core of my network. Loopback0 is advertised into this area as it will be used for the GRE tunnel source/destination.

HME-ASA1

interface Vlan10
 nameif mgmt
 security-level 100
 ip address 10.10.10.1 255.255.255.0
!
 interface Vlan101
  nameif gre-transit
  security-level 100
  ip address 172.168.255.17 255.255.255.240
!
 route-map CONN->OSPF permit 10
 match interface mgmt
!
 router ospf 1
  router-id 172.168.255.1
  network 172.168.255.16 255.255.255.240 area 0
 redistribute connected subnets route-map CONN->OSPF

HME-ASA1# show ospf int brief
Interface    PID   Area            IP Address/Mask    Cost State Nbrs F/C
gre-transit 1 0 172.168.255.17/255.255.255.240 10 DR 1

HME-ASA1# show ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.201.1 1 FULL/BDR 0:00:34 172.168.255.18 gre-transit

HME-CSR1

router ospf 1
  router-id 192.168.201.1
!
 interface GigabitEthernet1
  ip address 172.168.255.18 255.255.255.240
  ip ospf 1 area 0
!
 interface Loopback1
  ip address 192.168.201.1 255.255.255.255
  ip ospf 1 area 0

HME-CSR1# show ip ospf int bri
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo1          1     0               192.168.201.1/32   1     LOOP  0/0
Gi1          1     0               172.168.255.18/28  1     BDR   1/1

HME-CSR1# show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
172.168.255.1     1   FULL/DR         00:00:36    172.168.255.17  GigabitEthernet1

3. ASA IPsec configuration

The IPsec VPN interesting traffic only needs to carry the CSR loopback interfaces that will be used as the source and destination by the GRE tunnel.

Interesting traffic

object-group network HME-to-AZE-LOCAL
 network-object host 192.168.201.1
object-group network HME-to-AZE-REMOTE
 network-object host 172.168.201.1
!
access-list VPN-HME-to-AZE extended permit ip object-group HME-to-AZE-LOCAL object-group HME-to-AZE-REMOTE

If sysopt connection permit-vpn is disabled a permit rule is also needed in the outside ACL.

access-list outside extended permit ip object-group HME-to-AZE-REMOTE object-group HME-to-AZE-LOCAL

Will need to noNAT the traffic if it is matched by another NAT such as PAT. Easy way to check is with packet-tracer.

packet-tracer input gre-transit udp 192.168.201.1 500 172.168.201.1 500

Is best to put this rule near the top of the NAT rules as you don’t want this traffic to be accidentally matched by another NAT in the future.

nat (gre-trasnit,outside) source static HME-to-AZE-LOCAL HME-to-AZE-LOCAL destination static HME-to-AZE-REMOTE HME-to-AZE-REMOTE

Phase1 and Phase2 authentication and encryption algorithms

crypto ikev1 policy 10
 authentication pre-share
 encryption aes-256
 hash sha
 group 5
 lifetime 86400
crypto ikev1 enable outside
!
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto map outside_map 10 match address VPN-HME-to-AZE
crypto map outside_map 10 set pfs group5
crypto map outside_map 10 set peer <azure-public-ip>
crypto map outside_map 10 set ikev1 transform-set ESP-AES-256-SHA
crypto map outside_map interface outside
!
tunnel-group <azure-public-ip> type ipsec-l2l
tunnel-group <azure-public-ip> ipsec-attributes
 ikev1 pre-shared-key <myvpnisnotsecure>

4. Azure AZE-CSR1 base and IPsec configuration

The Internet facing interface of the CSR will be in its own VRF to ensure there is isolation between the Internet and the private network. This also allows a default route to be advertised over OSPF making all traffic in this network segment at the remote site uses the GRE tunnel whilst ensuring it doesn’t affect the GRE control traffic.
As the loopback is used to source the GRE tunnel that traverses the Internet it will also have to in this VRF.
Normally the public facing interface on the CSR would be statically assigned, in Azure this has to be DHCP.

vrf definition INET
 rd 1:0
 address-family ipv4
!
interface GigabitEthernet1
 vrf forwarding INET
 ip address dhcp
!
interface Loopback1
 vrf forwarding INET
 ip address 172.168.201.1 255.255.255.255
!
ip route vrf INET 0.0.0.0 0.0.0.0 172.60.99.1

Interesting traffic

object-group ip access-list extended VPN-HME-to-AZE
 permit ip host 172.168.201.1 host 192.168.201.1

Phase1 and Phase2 authentication and encryption algorithms

crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 5
!
crypto isakmp keepalive 10 2
crypto ipsec transform-set ESP-AES-256-SHA esp-aes 256 esp-sha-hmac
crypto map INET 1 ipsec-isakmp
 set peer <home-public-ip>
 set transform-set ESP-AES-256-SHA
 set pfs group5
 match address VPN-HME-to-AZE
!
interface GigabitEthernet1
 crypto map INET

As a VRF is being used for the transport a keyring is required for the pre-shared key.

crypto keyring FG-to-DC1 vrf INET
 pre-shared-key address <home-public-ip> 255.255.255.255 key <myvpnisnotsecure>

The CSR required the DPD keepalive command where as the ASA had this by default. The least this can be set to is an interval of 10 seconds and retry twice. So in theory if the VPN had issues and was stuck in a state it would take a max of 29 seconds for this to cleared.

The tunnel can now be brought up by pinging across it, should see encapsulated and decapsulated packets

AZE-CSR1# ping vrf INET 192.168.201.1 source loop1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.201.1, timeout is 2 seconds:
Packet sent with a source address of 172.168.201.1
!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 17/17/19 ms

AZE-CSR1# show crypto session detail
Crypto session current status
Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect

Interface: GigabitEthernet1
Uptime: 00:32:07
Session status: UP-ACTIVE
Peer: 87.115.228.25 port 4500 fvrf: INET ivrf: INET
Phase1_id: 172.168.255.1
Desc: (none)
Session ID: 0
IKEv1 SA: local 172.60.99.4/4500 remote 87.115.228.25/4500 Active
Capabilities:DKN connid:1049 lifetime:23:27:51
IPSEC FLOW: permit ip host 172.168.201.1 host 192.168.201.1
Active SAs: 2, origin: crypto map
Inbound:  #pkts dec'ed 573576 drop 0 life (KB/Sec) 4607909/1672
Outbound: #pkts enc'ed 511359 drop 0 life (KB/Sec) 4607946/1672

HME-ASA1# show vpn-sessiondb l2l
Session Type: LAN-to-LAN

Connection : 51.144.156.155
Index : 33 IP Addr : 51.144.156.155
Protocol : IKEv1 IPsecOverNatT
Encryption : IKEv1: (1)AES256 IPsecOverNatT: (1)AES256
Hashing : IKEv1: (1)SHA1 IPsecOverNatT: (1)SHA1
Bytes Tx : 61476 Bytes Rx : 61600
Login Time : 10:31:53 GMT Sun Nov 26 2017
Duration : 0h:36m:09s

5. GRE Tunnels

Now the IPsec tunnel is up just need to build the GRE tunnel on the Home CSR and the Azure CSR using the loopbacks configured earlier and run OPSF area 10 over this.

To account for the GRE (24 bytes) and ESP (upto 75 bytes dependant on algorithms used) overhead it is recommended to lower the MTU to 1400 bytes and TCP Maximum Segment Size (MSS) to 1360 bytes (IP header 20 Bytes, TCP/UDP header 20 Bytes). This will help to ensure that the end devices do the fragmentation rather than the network devices having to do it for every packet.

The Crypto traffic of the tunnel is to be in the INET VRF yet the tunnel itself is in the default global VRF. This is accomplished with the command tunnel vrf INET that specifies the tunnel crypto traffic will be through this VRF.

GRE keepalives are used to dynamically bring down the tunnel interface if the remote end is down. These timers are set to match the IPsec DPD keepalives as the GRE tunnel relies on the IPsec tunnel as its mode of transport.

AZE-CSR1

router ospf 1
 router-id 192.168.201.1
 passive-interface GigabitEthernet2
!
interface GigabitEthernet2
 ip address dhcp
 ip ospf 1 area 10
!
interface Tunnel10
 ip address 172.168.220.2 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 ip ospf hello-interval 3
 ip ospf 1 area 10
 keepalive 10 2
 tunnel source Loopback1
 tunnel destination 192.168.201.1
 tunnel vrf INET

HME-CSR1

interface Tunnel10
 ip address 172.168.220.1 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 ip ospf hello-interval 5
 ip ospf 1 area 10
 keepalive 10 2
 tunnel source Loopback1
 tunnel destination 172.168.201.1

Home CSR verification

HME-CSR1# show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
172.168.255.1     1   FULL/DR         00:00:39    172.168.255.17  GigabitEthernet1
172.168.201.1     0   FULL/  -        00:00:17    172.168.220.2   Tunnel10

HME-CSR1# show ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo1          1     0               192.168.201.1/32   1     LOOP  0/0
Gi1          1     0               172.168.255.18/28  1     BDR   1/1
Tu10         1     10              172.168.220.1/30   1000  P2P   1/1

HME-CSR1# show ip route ospf
Gateway of last resort is 172.168.255.17 to network 0.0.0.0
O*E1  0.0.0.0/0 110/12 via 172.168.255.17, 04:48:39, GigabitEthernet1
      10.0.0.0/24 is subnetted, 1 subnets
O E1     10.10.10.0 110/12 via 172.168.255.17, 04:48:39, GigabitEthernet1
      172.60.0.0/24 is subnetted, 1 subnets
O        172.60.1.0 110/1001 via 172.168.220.2, 04:08:31, Tunnel10
      172.168.0.0/16 is variably subnetted, 5 subnets, 3 masks
O        172.168.255.0/28
           110/11 via 172.168.255.17, 04:48:39, GigabitEthernet1

Azure CSR verification

AZE-CSR1# show ip ospf int brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Gi2          1     10              172.60.1.4/24      1     DR    0/0
Tu10         1     10              172.168.220.2/30   1000  P2P   1/1

AZE-CSR1# show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.201.1     0   FULL/  -        00:00:18    172.168.220.1   Tunnel10

AZE-CSR1#show ip route ospf
Gateway of last resort is 172.168.220.1 to network 0.0.0.0
O*E1  0.0.0.0/0 110/1012 via 172.168.220.1, 04:11:20, Tunnel10
      10.0.0.0/24 is subnetted, 5 subnets
O E1     10.10.10.0 110/1012 via 172.168.220.1, 04:11:20, Tunnel10
      172.168.0.0/16 is variably subnetted, 3 subnets, 3 masks
O IA     172.168.255.16/28 110/1001 via 172.168.220.1, 04:11:20, Tunnel10
      192.168.201.0/32 is subnetted, 1 subnets
O IA     192.168.201.1 110/1001 via 172.168.220.1, 04:11:20, Tunnel10

6. Quality of Service and Zone-Based Firewall

Enabling qos pre-classify under the tunnel interface would ensure that the TOS byte is passed onto the ESP packets, however it can only be used if IPsec and GRE tunnels terminate on the same device.
As QoS only kicks in if the interface is congested an alternative method is to use a child policy that allows you to cause congestion at less that the interfaces physical speed. The below example shapes at 100Mbps and then applies the QoS as per the child policy.

policy-map PM-CHILD
 class CONTROL
  bandwidth percent 5 
 class class-default
!
policy-map PM-PARENT
 class class-default
  shape average percent 100   
   service-policy PM-CHILD
!
int tun 10
 bandwidth qos-reference 1000000
 service-policy output PM-PARENT

As the Azure CSR is facing the Internet it needs locking down in some manner. Could use Azure furewall rules or the CSR ZBFW to ensure that the only traffic the CSR will accept is IPsec and GRE from the remote peer. Although it is a bit clunky to configure, as it is unlikely to change much the CSR ZBFW is a good place to do the packet filtering so that all the configuration is in the one place.

ip access-list extended VRF_OUTSIDE->SELF
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any port-unreachable
 permit icmp any any time-exceeded
 remark IPSEC TRAFFIC
 permit esp host <home-public-ip> host <azure-public-ip>
 permit udp host <home-public-ip> host <azure-public-ip> eq isakmp
 permit udp host <home-public-ip> host <azure-public-ip> eq 4500
 remark GRE TRAFFIC
 permit gre host 192.168.201.1 host 172.168.201.1
!
class-map type inspect match-all CM-VRF_OUTSIDE->SELF
 match access-group name VRF_OUTSIDE->SELF
!
policy-map type inspect PM-VRF_OUTSIDE->SELF
 class type inspect CM-VRF_OUTSIDE->SELF
  inspect
 class class-default
  drop
!
zone security VRF_OUTSIDE
!
zone-pair security ZP_VRF_OUTSIDE->SELF source VRF_OUTSIDE destination self
 service-policy type inspect PM-VRF_OUTSIDE->SELF
!
ip access-list extended SELF->VRF_OUTSIDE
 permit icmp any any
 permit udp any any range 33434 33464
 remark IPSEC TRAFFIC
 permit esp host <azure-public-ip> host <home-public-ip>
 permit udp host <azure-public-ip> host <home-public-ip> eq isakmp
 permit udp host <azure-public-ip> host <home-public-ip> eq 4500
 remark GRE TRAFFIC
 permit gre host 172.168.201.1 host 192.168.201.1
!
class-map type inspect match-all CM-SELF->VRF_OUTSIDE
 match access-group name SELF->VRF_OUTSIDE
!
policy-map type inspect PM-SELF->VRF_OUTSIDE
 class type inspect CM-SELF->VRF_OUTSIDE
  inspect
 class class-default
  drop
!
zone-pair security ZP_SELF->VRF_OUTSIDE source self destination VRF_OUTSIDE
 service-policy type inspect PM-SELF->VRF_OUTSIDE
!
int gi 1
 zone-member security VRF_OUTSIDE