Automate Leaf and Spine Deployment - Part5

The 5th post in the ‘Automate Leaf and Spine Deployment’ series goes through the deployment of the services that run on top of the fabric. These services are grouped into 3 categories, tenant, interface and routing. Services are configured only on the leaf and border switches, the spines have no need for them as they just route the VXLAN encapsulated packets with no knowledge or care of what is within them.

• Automate Leaf and Spine Deployment Part1 - introduction and structure
• Automate Leaf and Spine Deployment Part2 - input variable validation
• Automate Leaf and Spine Deployment Part3 - fabric variables and dynamic inventory
• Automate Leaf and Spine Deployment Part4 - deploying the fabric with ansible
• Automate Leaf and Spine Deployment Part5 - fabric services: tenant, interface, route
• Automate Leaf and Spine Deployment Part6 - post validation
• github repo

All the services are very similar in their structure in that they take input from a variable file and pass it through a filter plugin to create a data model that defines the service. Dependant on the service it could either be a per-device-type or per-device data model. All three services are for want of a better word, sub-roles within the service role. They each have separate variable files, tasks and templates as well as separate methods within the one filter plugin. The following services are built by each sub-role:

• tenant: VRFs, L3VNIs, L2VNIs, SVIs and VLANs
• interface: Loopbacks, single-homed (layer3, access, trunk) and dual-homed (access, trunk) interfaces
• routing: BGP, OSPF, static routes and redistribution

Within the main playbook each sub-role task is imported from the services role with tags associated in the playbook applied to all sub-role tasks.

    - name: Builds the tenant config snippets
      import_role:
        name: services
        tasks_from: svc_tnt
      tags: [tnt, bse_fbc_tnt, bse_fbc_tnt_intf, full]
    - name: Builds the interface config snippets
      import_role:
        name: services
        tasks_from: svc_intf
      tags: [intf, bse_fbc_tnt_intf, full]
    - name: Builds the tenant routing config snippets
      import_role:
        name: services
        tasks_from: svc_rte
      tags: [rte, full]

The following sections go through the possible setting (variables) in each sub-role before finishing with info on deployment details and tags.

Table Of Contents

Tenant (svc_tnt)

Tenants, SVIs, VLANs and VXLANs are created based on the variables stored in the service_tenant.yml file (svc_tnt.tnt).

tnt: A list of tenants that contains a list of VLANs (Layer2 and/ or Layer3)

• Tenants (VRFs) will only be created on a leaf or border if a VLAN within that tenant is to be created on that device
• Even if a tenant is not a layer3 tenant a VRF will still be created and the L3VNI and tenant VLAN number reserved
• If the tenant is a layer3 tenant the route-map for redistribution is always created and attached to the BGP peer

vlans: A List of VLANs within a tenant which at a minimum need the layer2 values of name and num. VLANs and SVIs can only be created on all leafs and/ or all borders, you can’t selectively say which individual leaf or border switches to create them on

• Unless an IP address is assigned to a VLAN (ip_addr) it will only be L2 VLAN
• L3 VLANs are automatically redistributed into BGP. This can be disabled (ipv4_bgp_redist: False) on a per-vlan basis
• By default VLANs will only be created on the leaf switches (create_on_leaf). This can be changed on a per-vlan basis to create only on borders (create_on_border) or on both leafs and borders
• To add a non-VXLAN SVI (without anycast address) create the VLAN as normal but with the extra VXLAN: False dictionary. The SVI is defined in service_interface.yml as type: svi
• Optional settings will implicitly use the default value, they only need defining if not using the default value

The redistribution route-map name can be changed in the advanced (adv) section of services-tenant.yml or services-routing.yml. If defined in both places the setting in services-routing.yml take precedence.

L2VNI and L3VNI numbers

The L2VNI and L3VNI values are automatically derived and incremented on a per-tenant basis based on the start and increment seed values defined in the advanced section (svc_tnt.adv) of services_tenant.yml.

adv.bse_vni: Starting VNI numbers

adv.vni_incre: Number by which VNIs are incremented for each tenant

For example a two tenant fabric each with a VLAN 20 using the above values would have L3 tenant SVIs of 3001, 3002, L3VNIs or 10003001, 10003002 and L2VNIs of 10020 and 20020.

Tasks and templates

A new data-model is created from the services_tenant.yml variables by passing them through the filter_plugin format_dm.py method create_svc_tnt_dm along with the BGP route-map name (if exists) and ASN (from fabric.yml). The result is a per-device-type (leaf and border) list of tenants, SVIs and VLANs.

- name: "Create the tenant configuration snippets"
  block:
    - name: "TNT >> Creating per-device type service_tenant data-models"
      set_fact:
        flt_svc_tnt: "{{ svc_tnt.tnt |create_svc_tnt_dm(svc_tnt.adv, fbc.adv.mlag.peer_vlan, svc_rte.adv.redist.rm_name
                        |default(svc_tnt.adv.redist.rm_name)) }}"
      changed_when: False
  check_mode: False
  run_once: true

This task only needs to be run once as the data models are created purely from the variable files, not from any of the host_vars. Below is an example of the data model format for a tenant and its VLANs.

{
    "bgp_redist_tag": 99,
    "l3_tnt": true,
    "l3vni": 100003004,
    "rm_name": "RM_CONN->BGP65001_RED",
    "tnt_name": "RED",
    "tnt_redist": true,
    "tnt_vlan": 3004,
    "vlans": [
        {
            "create_on_border": true,
            "create_on_leaf": false,
            "ip_addr": "10.99.99.1/24",
            "ipv4_bgp_redist": true,
            "name": "red_inet_vl99",
            "num": 99,
            "vni": 40099
        },
        {
            "ip_addr": "l3_vni",
            "ipv4_bgp_redist": false,
            "name": "RED_L3VNI",
            "num": 3004,
            "vni": 100003004
        }
    ]
}

This new data model (flt_svc_tnt) is used to render the svc_tnt_tmpl.j2 template and create the config snippet which will eventually be joined with bse_fbc and deployed to the devices. The template task conditionally only runs on border and leaf switches.

- name: "TNT >> Generating service_tenant config snippets"
  template:
    src: "{{ ansible_network_os }}/svc_tnt_tmpl.j2"
    dest: "{{ ans.dir_path }}/{{ inventory_hostname }}/config/svc_tnt.conf"
  changed_when: False
  check_mode: False
  when: bse.device_name.spine not in inventory_hostname

The same template is used for tenant creation on both the leaf and border switches. The conditional statement at the start of the template dictates whether the leaf or border data model (element 0 or 1 in the list) is iterated through by setting the value of the variable flt_vars dependant on the device it is being run against.

{% if bse.device_name.leaf in inventory_hostname %}{% set flt_vars = flt_svc_tnt[0] %}
{% elif bse.device_name.border in inventory_hostname %}{% set flt_vars = flt_svc_tnt[1] %}
{% endif %}

Interface (svc_intf)

The service_interface.yml variables define single or dual-homed interfaces (including port-channel) either statically or dynamically.

• By default all interfaces are dual-homed LACP ‘active’. The VPC number can not be changed, is always the port-channel number
• Interfaces and port-channels can be assigned dynamically from a pre-defined pool (under svc_intf.adv) or specified manually
• If the tenant (VRF) is not defined for a layer3, SVI or loopback interface it will be created in the global routing table
• If the interface config is the same across multiple switches (like an access port) define one interface with a list of switches
• Only specify the odd numbered switch for dual-homed interfaces, the config for MLAG neighbor is automatically generated

There are 7 pre-defined interface types that can be deployed:

• access: A single VLAN layer2 access port with STP set to ‘edge’
• stp_trunk: A trunk going to a device that supports Bridge Assurance. STP is set to ‘network’
• stp_trunk_non_ba: Same as stp_trunk except that STP is set to ‘normal’ as it is for devices that don’t support BA
• non_stp_trunk: A trunk port going to a device that doesn’t support BPDU. STP is set to ‘edge’ and BPDU Guard enabled
• layer3: A layer3 interface with an IP address. Must be single-homed as MLAG not supported for L3 interfaces
• loopback: A loopback interface with an IP address (must be single-homed)
• svi: To define a SVI the VLAN must exist in service_tenant.yml and not be a VXLAN (must be single-homed)

The intf.single_homed and intf.dual-homed dictionaries hold a list of all single-homed or dual-homed interfaces using any of the attributes in the table below. If there are no single-homed or dual-homed interfaces on the fabric hash out the relevant dictionary.

The playbook has the logic to recognize if statically defined interface numbers overlap with the dynamic interface range and exclude them from dynamic interface assignment. For simplicity it is probably best to use separate ranges for the dynamic and static assignments.

adv.single_homed: Reserved range of interfaces to be used for dynamic single-homed and loopback assignment

adv.dual-homed: Reserved range of interfaces to be used for dynamic dual-homed and port-channel assignment

Tasks and templates

The format_dm.py filter_plugin method create_svc_intf_dm is run for each inventory host to produce a list of all interfaces to be created on that device. In addition to the service_interface.yml variables it also passes in the interface naming format (fbc.adv.bse_intf) to create the full interface name and hostname to find the interfaces relevant to that device.

- name: "Create the interface configuration snippets"
  block:
    - name: "SYS >> Creating per-device service_interface data-models"
      set_fact:
        flt_svc_intf: "{{ svc_intf.intf |create_svc_intf_dm(inventory_hostname, svc_intf.adv, fbc.adv.bse_intf) }}"
      changed_when: False
  check_mode: False

Below is an example of the data model format for a single-homed and dual-homed interface.

{
    "descr": "UPLINK > DC1-BIP-LB01 - Eth1.1",
    "dual_homed": false,
    "intf_num": "Ethernet1/9",
    "ip_vlan": 30,
    "stp": "edge",
    "type": "access"
},
{
    "descr": "UPLINK > DC1-SWI-BLU01 - Gi0/0",
    "dual_homed": true,
    "intf_num": "Ethernet1/18",
    "ip_vlan": "10,20,30",
    "po_mode": "on",
    "po_num": 18,
    "stp": "network",
    "type": "stp_trunk"
},
{
    "descr": "UPLINK > DC1-SWI-BLU01 - Po18",
    "intf_num": "port-channel18",
    "ip_vlan": "10,20,30",
    "stp": "network",
    "type": "stp_trunk",
    "vpc_num": 18
}

As with service_tenant the fact flt_svc_intf is used to render the template and produce the configuration snippet. The template is quite simple as it is the exact same format to create interfaces for all devices types.

Route (svc_rte)

BGP peerings, non-backbone OSPF processes, static routes and redistribution (connected, static, bgp, ospf) are configured based on the variables specified in the service_route.yml file. The naming convention of the route-maps and prefix-lists used by OSPF and BGP can be changed under the advanced section (adv) of the variable file.

I am undecided about this role as it goes against the simplistic principles defined in the design goals and used by other roles.

Simple yet functional – Keep it as simple as possible without sacrificing too much functionality. Weigh up whether it is worth allowing a feature to be customizable vs the extra complexity this adds

By its very nature routing is very configurable which leads to complexity due to the number of options and inheritance. I have tried to include the things that I think would most often be used, however there are still things like tags and communities that I decided not to add as they kept taking me deeper and deeper down a rabbit hole of complexity.

In theory all these features should work but due to the number of options and combinations available I have not tested all the possible variations of configuration.

Static routes (svc_rte.static_route)

Routes are added per-tenant with the tenant being the top-level dictionary that routes are created under.

• tenant, switch and prefix are lists to make it easy to apply the same routes across multiple devices and tenants
• For routes with the same attributes (like next-hop) can group all the routes as a list within the one prefix dictionary value
• Can optionally set next-hop interface, administrative distance and the next hop VRF (for route leaking between VRFs)

OSPF (svc_rte.ospf)

An OSPF processes can be configured for any of the tenants or the global routing table. Each OSPF process is enabled on a per-interface basis with summarization and redistribution defined on a per-switch basis.

• The mandatory process.switch list defines the switches the OSPF process is enabled on. Router-IDs (RID) are optional, if configured they must match the exact number of switches (1-to-1 relationship)
• By default passive interface is globally enabled and disabled on all interfaces. This can be enabled on a per-interface basis
• If authentication is enabled for any one interface it is enabled globally for that whole area
• If BFD is enabled globally it can be disabled on a per-interface basis by setting the OSPF timers for that interface
• Non-mandatory settings only need to be defined if changing the default behavior, otherwise is no need to add the dictionary

Interface, summary and redistribution are child dictionaries of lists under the ospf parent dictionary. They inherit process.switch unless switch is specifically defined under that child dictionary.

ospf.interface: Each list element is a group of interfaces with the same set of attributes (area number, interface type, auth, etc)

ospf.summary: All summaries with the same attributes (switch, filter, area) can be grouped in a list within the one prefix dictionary value

ospf.redist: Each list element is the redistribution type, can be ospf xx, bgp xx, static or connected

• Redistributed prefixes can be filtered (allow) or weighted (metric). If the allow list is not set is allow any (empty route-map)
• The switch set under the redistribution type is preferred over that set in process.switch, there is no merging
• Can only have one of each of the redistribution types connected and static per-switch. Only the first occurrence is used, with the redistribution type switch always preferred over process.switch

BGP (svc_rte.bgp)

Uses the concept of groups and peers with the majority of the settings configured in either. Peer settings take precedence over group

• group holds the global settings for all peers within it. Are automatically created on any switches that peers within it are created
• peer is a list of peers within the group. If the setting is configured in the group and peer the peer setting will take precedence
• The group.name and peer.name are used in the construction of route-map and prefix-list names
• switch is a list of switches (even if single device) to allow for the same group and peers to be created on multiple devices
• tenant is a list of VRFs (even if a single VRF) to allow for the same peers to be created in multiple tenants. If the tenant is not specified (dictionary not defined) the group or peer will be added to the default global routing table
• Keywords any and default can be used instead of a list of prefixes with filtering (inbound/outbound) and redistribution
• Non-mandatory settings only need to be defined if changing the default behavior, otherwise is no need to add the dictionary

inbound or outbound: Optionally set under the group or peer to filter BGP advertisements and/ or BGP attribute manipulation

• The naming of the route-maps and prefix-lists are dependant on where they are applied (group or peer)
• All BGP attribute settings are dictionaries with the key being the attribute and the value the prefixes it is applied to
• as_prepend and med can be applied to outbound advertisements with weight and pref (local preference) applied to those received inbound. If everything is defined the route-map sequence order is BGP_ATTR, deny_specific, allow_specific, allow_all, deny_all

bgp.tnt_advertise: Optionally advertise prefixes on a per-tenant basis (list of VRFs) using network, summary and redistribution. The switch can be set globally for all network/summary/redistribution in a VRF and be overridden on an individual per-prefix basis

• network: List of prefixes to be advertised on a per-switch basis (network cmd). If a device is covered by 2 different network.prefix statements it will get a combination of them both (merged), so network statements for all prefixes
• summary: Group summaries (aggregate-address) with the same attributes (switch and summary_only) within the same list element
• redist: Each list element is the redistribution type (ospf process, static or connected) with the redistributed prefixes weighted (metric) and/or filtered (allow). If the allow list is not set it is allow any (empty route-map). Can only have one each of types connected and static per-switch, the first occurrence is used. The switch set under the redistribution type is preferred over that set in process.switch, is no merging

Advanced (svc_rte.adv)

Advanced settings allow the changing of the default routing protocol timers and naming convention of the route-maps and prefix-lists used for advertisement and redistribution. The only naming restrictions are the keywords that are used in the filter plugins logic to swap the keyword for another values.

Tasks and templates

The filter_plugin method create_svc_rte_dm is run for each inventory device to produce a data model of the routing configuration for that device. Deploying any of the routing options (BGP, BGP tenant advertisement, OSPF, static routes) is optional so has to be fed into the plugin separately with the Ansible default filter so as not to fail if it were empty.

- name: "Create the routing configuration snippets"
  block:
    - name: "SYS >> Creating per-device service_route data-models"
      set_fact:
        flt_svc_rte: "{{ inventory_hostname |create_svc_rte_dm(svc_rte.bgp.group |default (), svc_rte.bgp.tnt_advertise |default (),
                         svc_rte.ospf |default (), svc_rte.static_route |default (), svc_rte.adv, fbc) }}"
      changed_when: False
  check_mode: False

The method used for creating the routing data models is a lot longer and more complex than service_tenant or service_interface methods. To try and avoid repetitive code a lot of the complexity around building prefix-lists and route-maps data models is put into separate methods.

• create_bgpattr_rm_pfx_lst: The prefix-list and route-map for BGP attribute associated prefixes (weight, local pref, med & AS-path)
• create_allowdeny_rm_pfx_lst: Prefix-list and route-map (order deny/allow_spec, allow_any/deny_any) for allowed & denied prefixes
• create_redist_rm_pfx_lst: The prefix-list and route-map for prefix redistribution and MED/metric manipulation

The outcome is a list of seven per-device data models that are used by the svc_rte_tmpl.j2 template.

• all_pfx_lst: List of all prefix-lists with each element in the format [name, seq, permission, prefix]
• all_rm: List of all route-maps with each element in the format [name, seq, permission, prefix, [attribute, value]]. If no BGP attributes are set in the RM the last entry in the list will be [null, null]
• stc_rte: Per-VRF dictionaries (VRF is the key) of lists of static routes with interface and/or gateway, optional AD and destination VRF
• group: Dictionaries of BGP groups (group is the key) that have peers on this device. The value is dictionaries of any group settings
• peer: Dictionaries of tenants (VRFs) containing the following nested dictionaries:
  • peers: Dictionary of peers (key is the peer) with the value being dictionaries of the peers settings
  • network: List of networks to be advertised by BGP
  • summary: Dictionary of summaries with the key being the prefix and value either null (doesn’t suppress) or summary-only
  • redist: Two dictionaries of the route-map name (rm_name) and redistribution type (connected, static, etc)
• ospf_proc: Dictionary of VRFs (key) and the OSPF process settings for each VRF (settings configured under the process)
• ospf_intf: Dictionary of interfaces (key) that have OSPF enabled, the values are the interface specific OSPF settings

Running the roles

Due to the declarative nature of the playbook and inheritance between roles there are only a certain number of combinations that the roles can be deployed in. It can take 3 to 4 minutes to deploy the full configuration to NXOS when including the service roles so the Napalm default timeout has been increased to 240 seconds.

• diff tag can be used with bse_fbc_tnt, bse_fbc_intf, full or rb to print the configuration changes to screen
• Changes are always saved to file no matter whether diff is used or not
• -C or –check-mode will do everything except actually apply the configuration

pre-validation: Validates the contents of variable files defined under var_files. Best to use dummy host file instead of dynamic inventory

ansible-playbook PB_build_fabric.yml -i hosts --tag post_val

Generate the complete config: Creates config snippets, assembles them in config.cfg, compares against device config and prints the diff

ansible-playbook PB_build_fabric.yml -i inv_from_vars_cfg.yml --tag 'ful, diff' -C

Apply the config: Replaces current config on the device with changes made automatically saved to ~/device_configs/diff/device_name.txt

ansible-playbook PB_build_fabric.yml -i inv_from_vars_cfg.yml --tag full

All roles can be deployed individually to just to create the config snippet files, no connections are made to devices or changes applied. The merge tag can be used in conjunction with any combination of these role tags to non-declaratively merge the config snippets with the current device configuration rather than replacing it. As the L3VNIs and interfaces are generated automatically at a bare minimum the variable files will still need current tenants and interfaces as well as the advanced variable sections.

ansible-playbook PB_build_fabric.yml -i inv_from_vars_cfg.yml --tag 'tnt,intf,rte'

ansible-playbook PB_build_fabric.yml -i inv_from_vars_cfg.yml --tag tnt,rte,merge,diff

• ← Previous Post
• Next Post →