NSX IPSec VPN between datacenters (multi site/region)

I’m doing some lab work with my team at the moment and we were gifted some hardware to do some multi region validation. Both systems (a VxRack SDDC & a VxRail) are in 2 separate datacenters, and both are using private IP addressing that is not routable between datacenters. As part of the validation we need both systems to be able to communicate with each other, however we dont control the inter lab switching to put in place the necessary routes to enable this. Rather than go through a change control process with the keepers of that gate we decided to get creative and have some fun (and hopefully learn something!) by setting up an NSX IPSec VPN between the labs.

Disclaimer: There are many better ways to do this for a permanent lab setup (i.e. BGP to the core with routes) but this was done on borrowed kit that was never initially designed with inter lab routing as a requirement, with no direct control on the inter lab switches, and we would also like to put it back the way we found it so dont want to make sweeping architectural changes!

The configuration we were given looked like this initially:

 

• Lab A & Lab B are 2 independent datacenters
• There is an Inter Lab Core between them (controlled by a higher power!)
• There is no routing in place to get from Lab A subnet 172.19.5.0/25 to Lab B subnet 172.25.105.0/24
• The gateway for Lab A subnet 172.19.5.0/25 is a Switch Virtual Interface (SVI) on Lab A 7K
• The gateway for Lab B subnet 172.25.105.0/24 is a Switch Virtual Interface (SVI) on Lab B 7K
• VC01 in Lab A cannot currently communicate with VC02 in Lab B
• NSX is deployed and configured in both Lab vCenters in standalone mode

 

To get the inter lab communications working we opted for an IPSec VPN, which essentially gives you a secure encrypted virtual private network between sites. To facilitate the VPN connectivity we deployed an NSX edge in each lab and configured some additional interfaces:

Lab A

• vNIC0 is an uplink to the inter-lab core on 219.222/24
• vNIC2 is an internal link to the private lab address space 172.19.5.39/25

Lab B

• vNIC0 is an uplink to the inter-lab core on 47.222/24
• vNIC1 is an internal link to the private lab address space 172.25.105.222/24

The setup is starting to take shape now and looks something like this

 

Once the interfaces are in place we can setup the IPSec VPN

1. Browse to the VPN tab on the edge in Lab A
2. First you need to add a pre shared key
   1. Under Global configuration status click Change
   2. Enter a Pre-Shared Key and click OK
3. Click Add to add a new IPSec VPN
   • Local Id: We used the local “public” IP address but this can be any friendly name. It will be entered on the target side as the peer Id
   • Local Endpoint: Local “public” IP address
   • Local Subnets: The local subnet(s) you want to be exposed as part of the VPN. In this case 172.19.5.0/25
   • Peer Id: We used the remote “public” IP address but this can be any friendly name. It will be entered on the target side as the local Id
   • Peer Endpoint: Remote “public” IP address
   • Peer Subnets: The remote subnet(s) you want to be connect to as part of the VPN. In this case 172.25.105.0/24
   • We kept the defaults for IKE version, Digest & Encryption Algorithm
   • Pre-Shared Key: Enter the Pre-Shared Key created in step 2.2 above
   • We kept the defaults for Diffie-Hellman Group

The resulting config should look something like this:

4. Click OK
5. Click Start & Click Publish

Repeat steps 1-5 on the target side ensuring you reverse the local & peer settings. Once this is done you should be able to click on Show IPSec Statistics on either side to view the Tunnel Status. If everything is configured correctly you should see the Tunnel Status is Up

Normally this would be enough if all VMs were using a gateway on these edges but if you refer back to the original setup section at the top of this post you will see the following points:

• The gateway for Lab A subnet 172.19.5.0/25 is a Switch Virtual Interface (SVI) on Lab A 7K
• The gateway for Lab B subnet 172.25.105.0/24 is a Switch Virtual Interface (SVI) on Lab B 7K

Ideally you would move the gateways from the switches to the edges, but as we want to make minimal changes to the existing setup we are leaving the SVIs in place. So to close the loop and activate all the routes we needed to add the following static routes on each Lab 7K to ensure traffic destined for cross lab targets knows to go to the edges and then gets routed across the sites via the IPSec VPN

Lab A 7K

• ip route 172.25.105.0 255.255.255.0 172.19.5.39

Lab B 7K

• ip route 172.19.5.0 255.255.255.128 172.25.105.222

And there you have it. A functioning NSX IPSec VPN between 2 datacenters/sites/regions!

Should out to Ken Gould, Ryan Whalley, Dave Lake & Dave Ryan for their assistance!