Create a multi pNIC VMware Cloud Foundation NSX-V Workload Domain with PowerVCF

Hopefully by now you’ve seen my earlier posts about the new PowerShell module for the VMware Cloud Foundation API. If not i’d suggest reviewing these before reading on

With the release of VMware Cloud Foundation 3.9.1 it is now supported, via the API only, to use more than 2 physical NICs (pNICs) per host. In fact the API now supports up to three vSphere Distributed switches and six physical NICs, providing more flexibility to support high performance use cases and physical traffic separation.

There is a tech note that goes into more detail on the use cases for more than 2 pNICs and it also shows how this works using PostMan but we can also achieve this using PowerVCF.

The workflow using PowerVCF is the same as my earlier example for creating a workload domain. The only difference is the content in the JSON file.

Note: There is a validation API to validate the JSON you are passing before making the submission. PowerVCF dynamically formats the validation JSON as the formatting is slightly different to what you submit to create the workload domain.

To get you started there is a sample JSON file with the required formatting. Here is a snapshot of what it looks like

{
  "domainName": "PowerVCF",  
  "vcenterSpec": {  
    "name": "sfo01w01vc01",  
    "networkDetailsSpec": {  
       "ipAddress": "172.16.225.64",  
       "dnsName": "sfo01w01vc01.sfo01.rainpole.local",  
       "gateway": "172.16.225.1",  
       "subnetMask": "255.255.255.0"
     },  
     "rootPassword": "VMw@re1!",  
     "datacenterName": "PowerVCF-DC"  
   },  
   "computeSpec": {  
      "clusterSpecs": [ {  
          "name": "Cluster1",  
          "hostSpecs": [ {  
              "id": "d0693b58-4012-4387-92ed-721cfa709e44",
              "license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ]  
               }  
            }, {  
              "id": "7006bec4-fccb-49a0-bff6-fd56c807d26a",
              "license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ]  
               }  
            }, {  
              "id": "cc257a80-e179-4297-bf7e-179a0944bbab",
              "license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ] 
               } 
           } ],     
    "datastoreSpec": {  
        "vsanDatastoreSpec": {  
            "failuresToTolerate": 1,  
            "licenseKey": "BBBBB-BBBBB-BBBBB-BBBBB-BBBBB",
            "datastoreName": "vSanDatastore" 
         }  
     },  
     "networkSpec": { 
         "vdsSpecs": [ { 
             "name": "SDDC-Dswitch-Private1", 
             "portGroupSpecs": [ {  
                 "name": "SDDC-DPortGroup-Mgmt", 
                 "transportType": "MANAGEMENT" 
             }, { 
                 "name": "SDDC-DPortGroup-VSAN",  
                 "transportType": "VSAN" 
             }, {  
                 "name": "SDDC-DPortGroup-vMotion", 
                 "transportType": "VMOTION" 
             } ] 
          },  
          {  
             "name": "SDDC-Dswitch-Private2", 
             "portGroupSpecs": [ { 
                "name": "SDDC-DPortGroup-Public", 
                "transportType": "PUBLIC"  } ] 
           } 
        ],  
        "nsxClusterSpec": { 
           "nsxVClusterSpec": {  
              "vlanId": 2237,  
              "vdsNameForVxlanConfig": "SDDC-Dswitch-Private1"  
            }  
          }  
        }  
      } ] 
   }, 
  "nsxVSpec" : {
    "nsxManagerSpec" : {
      "name" : "sfo01w01nsx01",
      "networkDetailsSpec" : {
        "ipAddress" : "172.16.225.66",
        "dnsName" : "sfo01w01nsx01.sfo01.rainpole.local",
        "gateway" : "172.16.225.1",
        "subnetMask" : "255.255.255.0"
      }
    },
    "nsxVControllerSpec" : {
      "nsxControllerIps" : [ "172.16.225.121", "172.16.225.122", "172.16.225.123" ],
      "nsxControllerPassword" : "VMw@re123456!",
      "nsxControllerGateway" : "172.16.225.1",
      "nsxControllerSubnetMask" : "255.255.255.0"
    },
    "licenseKey" : "CCCCC-CCCCC-CCCCC-CCCCC-CCCCC",
    "nsxManagerAdminPassword" : "VMw@re1!",
    "nsxManagerEnablePassword" : "VMw@re1!"
  }
}

You can see that the magic happens in the hostNetworkSpec section where you map each vmnic to a vdsName

<p>"hostNetworkSpec": { "vmNics": [ { "id": "vmnic0", "vdsName": "SDDC-Dswitch-Private1" }, { "id": "vmnic1", "vdsName": "SDDC-Dswitch-Private1" }, { "id": "vmnic2", "vdsName": "SDDC-Dswitch-Private2" }, { "id": "vmnic3", "vdsName": "SDDC-Dswitch-Private2" } ] }</p>

So please try it out and let us know how it goes!

Create a new VMware Cloud Foundation Workload Domain with PowerVCF

So now that we have a PowerShell module for the VMware Cloud Foundation API, just what can we do with it? Well in this example we will create an NSX-V backed VMware Cloud Foundation workload domain, all using PowerVCF to interact with the API. Now all of this could obviously be wrapped up in a single script but I’m going to show you each step, with some tips along the way.

I will be making the assumption that you are familiar with VMware Cloud Foundation Concepts. If not please review the documentation here.

So once you have the initial VCF bringup completed you need to add a workload domain(s) to service our workloads. In my example below I have a management domain only.

And i have only the 4 hosts that are part of the management domain in my inventory. So i need to add new hosts to my inventory before i can create a new workload domain.

The sequence of events is as follows:

  • Install the PowerVCF Module
  • Connect to SDDC Manager
  • Create a network pool
  • Commission hosts
  • Create Workload domain

Install the PowerVCF Module from the PowerShell Gallery

  • Open PowerShell
  • Run the following to install the module

Install-Module -Name PowerVCF

Connect to SDDC Manager

  • To establish a session with SDDC Manager run the following

Connect-VCFManager -fqdn sddc-manager.sfo01.rainpole.local -username admin -password VMw@re1!

Create a network pool

The first thing you need before you can commission new hosts is to create a new network pool, which will include the vSAN & vMotion network details for this workload domain cluster.

To create a new network pool do the following:

  • Before you can create a network pool you first need to create the json body that will be passed in.

TIP: The PowerVCF Module includes a folder of sample json files to get you started

Here is the json format required for creating a vSAN network pool (Please use the same json with the module rather than copying from here as formatting is probably messed up!)


{
"name": "sfo01w01-cl01",
"networks": [
{
"type": "VSAN",
"vlanId": 2240,
"mtu": 9000,
"subnet": "172.16.240.0",
"mask": "255.255.255.0",
"gateway": "172.16.240.253",
"ipPools": [
{
"start": "172.16.240.5",
"end": "172.16.240.100"
}
]
},
{
"type": "VMOTION",
"vlanId": 2236,
"mtu": 9000,
"subnet": "172.16.236.0",
"mask": "255.255.255.0",
"gateway": "172.16.236.253",
"ipPools": [
{
"start": "172.16.236.5",
"end": "172.16.236.100"
}
]
}
]
}

So first off lets get a list of current Network Pools. To do this run the following cmdlet:

Get-VCFNetworkPool

As expected this returns a single network pool.

So to create a new network pool using the json you created earlier run the following:

New-VCFNetworkPool -json .\SampleJSON\NetworkPool\addNetworkPoolSpec.json

Now running Get-VCFNetworkPool should display 2 Network Pools

Commission Hosts

Now that you have a network pool you can commission hosts and associate them with the network pool. For this you need the following json

TIP: For this json you need the network pool name & ID. These were returned when the pool was created and also by Get-VCFNetworkPool


[
{
"fqdn": "sfo01w01esx01.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"
},
{
"fqdn": "sfo01w01esx02.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"

},
{
"fqdn": "sfo01w01esx03.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"
},
{
"fqdn": "sfo01w01esx04.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"
}
]

So to commission the 4 new hosts into my VCF inventory i simply run

Commission-VCFHost -json .\SampleJSON\Host\commissionHosts.json

TIP: This returns a task id, which you can monitor by running the following until status=Successful:

Get-VCFTask -id b93e2bc7-627b-4f7c-980b-c12b3497c4ea

Create a Workload Domain

Once the commission hosts task is complete you can then create a workload domain using those hosts. Creating a workload domain also requires a json file. For this you need the id’s of the hosts that you want to use. In VCF hosts that are available to be used in a workload domain have a status of UNASSIGNED_USEABLE so to find the id’s of the hosts you want to add run the following

TIP: Filter the results by adding | select fqdn,id

Get-VCFHost -Status UNASSIGNED_USEABLE | select fqdn,id

This returns the ids you need for creating the workload domain. Here is the Workload domain json. (Replace ESXi licence (AAAAA), vSAN licence (BBBBB) & NSX-V licence (CCCCC) with your keys)


{
"domainName" : "PowerVCF",
"vcenterSpec" : {
"name" : "sfo01w01vc01",
"networkDetailsSpec" : {
"ipAddress" : "172.16.225.64",
"dnsName" : "sfo01w01vc01.sfo01.rainpole.local",
"gateway" : "172.16.225.1",
"subnetMask" : "255.255.255.0"
},
"rootPassword" : "VMw@re1!",
"datacenterName" : "PowerVCF-DC"
},
"computeSpec" : {
"clusterSpecs" : [ {
"name" : "Cluster1",
"hostSpecs" : [ {
"id" : "dd2ec05f-39e1-464e-83f1-1349a0dcf723",
"license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
"hostNetworkSpec" : {
"vmNics" : [ {
"id" : "vmnic0",
"vdsName" : "sfo01w01vds01"
}, {
"id" : "vmnic1",
"vdsName" : "sfo01w01vds01"
} ]
}
}, {
"id" : "809b25e8-1db6-464b-b310-97f581c56da5",
"license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
"hostNetworkSpec" : {
"vmNics" : [ {
"id" : "vmnic0",
"vdsName" : "sfo01w01vds01"
}, {
"id" : "vmnic1",
"vdsName" : "sfo01w01vds01"
} ]
}
}, {
"id" : "5d3eea32-6464-4ae6-9866-932fb926a5f1",
"license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
"hostNetworkSpec" : {
"vmNics" : [ {
"id" : "vmnic0",
"vdsName" : "sfo01w01vds01"
}, {
"id" : "vmnic1",
"vdsName" : "sfo01w01vds01"
} ]
}
} ],
"datastoreSpec" : {
"vsanDatastoreSpec" : {
"failuresToTolerate" : 1,
"licenseKey" : "BBBBB-BBBBB-BBBBB-BBBBB-BBBBB",
"datastoreName" : "sfo01w01vsan01"
}
},
"networkSpec" : {
"vdsSpecs" : [ {
"name" : "sfo01w01vds01",
"portGroupSpecs" : [ {
"name" : "sfo01w01vds01-Mgmt",
"transportType" : "MANAGEMENT"
}, {
"name" : "sfo01w01vds01-VSAN",
"transportType" : "VSAN"
}, {
"name" : "sfo01w01vds01-vMotion",
"transportType" : "VMOTION"
} ]
} ],
"nsxClusterSpec" : {
"nsxVClusterSpec" : {
"vlanId" : 2237,
"vdsNameForVxlanConfig" : "sfo01w01vds01"
}
}
}
} ]
},
"nsxVSpec" : {
"nsxManagerSpec" : {
"name" : "sfo01w01nsx01",
"networkDetailsSpec" : {
"ipAddress" : "172.16.225.66",
"dnsName" : "sfo01w01nsx01.sfo01.rainpole.local",
"gateway" : "172.16.225.1",
"subnetMask" : "255.255.255.0"
}
},
"nsxVControllerSpec" : {
"nsxControllerIps" : [ "172.16.235.121", "172.16.235.122", "172.16.235.123" ],
"nsxControllerPassword" : "VMw@re123456!",
"nsxControllerGateway" : "172.16.235.1",
"nsxControllerSubnetMask" : "255.255.255.0"
},
"licenseKey" : "CCCCC-CCCCC-CCCCC-CCCCC-CCCCC",
"nsxManagerAdminPassword" : "VMw@re1!",
"nsxManagerEnablePassword" : "VMw@re1!"
}
}

To create the workload domain run the following:

New-VCFWorkloadDomain -json .\SampleJSON\WorkloadDomain\workloadDomainSpec-NSX-V.json

This will return a Task ID. Monitor the workload domain creation by running the following

Get-VCFTask -id b93e2bc7-627b-4f7c-980b-c12b3497c4ea

And that should be it. If you’ve gotten all your json details correct you should have a fully functioning NSX-V workload domain without using the UI!

Introducing PowerVCF – A PowerShell Module for the VMware Cloud Foundation API

Its been a while since I’ve posted something so I thought it was about time! Since joining VMware a year ago I’ve been heads down drinking from the firehose, learning from a phenomenal team and generally keeping very busy. More recently I’ve been playing a lot with VMware Cloud Foundation (VCF). A recent release (3.8) introduced a public API and I started getting field questions on how to leverage it so I started digging. The API has been expanded in 3.9. It is based on the OpenAPI standard (formerly Swagger) and can be accessed through the developer center in the SDDC Manager UI or via code.vmware.com

Now I’m not a developer so I fell back on Postman to do some initial testing. I like Postman as it dumbs it down for us non-devs 🙂 but I wanted something a little easier to consume so i started a little side project called PowerVCF (hat-tip to the far superior PowerNSX, PowerVRA, PowerVRO)

Basically I wanted to provide a simple, efficient, PowerCLI style experience for consuming the VMware Cloud Foundation public API.

Solution?

I am delighted to unleash the first iteration of PowerVCF on the community! Creating this has been a great learning experience for me. In the process I’ve improved my PowerShell skills, learned Git, Markdown and have started looking into CI/CD workflows. It’s also my first submission to the PowerShell Gallery.

Automate your VMware Validated Design NSX-V Distributed Firewall Configuration

A few weeks back I mentioned on twitter that i was working on automating the VMware Validated Design NSX-V Distributed Firewall Configuration in my lab. (I admit it took longer than i had planned!) Currently this is a manual post deployment step once VMware Cloud Builder has completed the deployment. This will likely be picked up by Cloud Builder in a future release but for now its a manual, and somewhat tedious, but required, step!

Full details on the manual steps required for this configuration can be found here. Please take the time to understand what these rules are doing before implementing them.

So in an effort to make this post configuration step a little less painful i set out to automate it. I’ve played with the NSX-V API in the past and found it much easier to interact with by using PowerNSX, rather than leveraging PostMan and the API directly. PowerNSX is the unofficial, official automation tool for NSX. Hats off to VMware engineers Nick Bradford, Dale Coghlan & Anthony Burke for creating and documenting this tool. Anthony also published a FREE book on Automating NSX for vSphere with PowerNSX. More on that here.

Disclaimer: This script is not officially supported by VMware. Use at your own risk & test in a development/lab environment before using in production.

I’ve posted the script to GitHub here as its a bit lengthy! There may be a more efficient way to do some parts of it and if anyone wants to contribute please feel free!

As with a lot of the scripts i create it is menu based and has 2 main options:

  1. Create DFW exclusions, IP Sets & Security Groups
  2. Create DFW Rules

The reason i split it into 2 distinct operations is to allow you to inspect the exclusion list, IP Sets & Security Groups before creating the firewall rules. This will ensure that you dont lock yourself out of vCenter by creating an incorrect rule.

Required Software

  • PowerCli
    • The script will check for PowerCli and if not found will attempt to install the latest version from the PowerShell Gallery
    • Currently tested on Windows only
    • If you dont have internet access you can manually install PowerCli by opening a PowerShell console as administrator and running:
    • Find-Module -Name VMware.PowerCLI | Install-Module
  • PowerNSX
    • The script will check for PowerNSX and if not found will attempt to install the latest version from the PowerShell Gallery
    • Currently tested on Windows only
    • If you dont have internet access you can manually install PowerNSX by opening a PowerShell console as administrator and running:
    • Find-Module -Name PowerNSX | Install-Module

Required Variables

Before you can run the script you need to edit the User Variables to provide the following:

  • Target vCenter details
    • Required to establish a PowerCli Connection with vCenter Server. This is used when updating the DFW exclusion list
  • Target NSX Manager details
    • Required to establish a connection with NSX manager to configure the DFW
  • IP Addresses for the various SDDC components

Hopefully you will find this useful!

What not to do when your Platform Services Controllers are Load Balanced!

I needed to do some validation around vRealize Operations Manager & vRealize Orchestrator for an upcoming VVD release and a physical lab environment was made available. The environment is a dual region VVD deployment. Upon verifying that I had access to all the components I needed it became obvious there was an issue with SSO in the primary region (SFO). Browsing to the web client for the SFO management vCenter I was seeing this:

As i mentioned this is a VVD deployment and per VVD guidelines there are 2 Platform Services Controllers (PSCs) behind an NSX load balancer per region. Like so: (Diagram from the VMware Validated Design 5.0 Architecture & Design guide)

Like any good (lazy!) IT person the first thing i did was google the error to find the quick fix! That led me to this communities post which had some suggestions around disk space etc. None of which were relevant to my issue. Running the following on the PSCs and vCenters showed that some services were not starting

service-control –status

Restarting the services didn’t help. Next up i checked the usual suspects:

  • NTP
  • DNS
  • SSL Certificates

All of the above looked ok. Next I turned my attention to the load balancer. Because the vCenter Web Client was inaccessible I was not able to access the load balancer settings through the UI so I turned to the NSX API using Postman

To connect to the NSX manager that is associated with the load balancer you need to configure a Postman session with basic authentication and enter the NSX manager admin user & password.

To retrieve information on the load balancer you need to run the following GET:

https://sfo01m01nsx01.sfo01.rainpole.local/api/4.0/edges/edge-1/loadbalancer/config

I wont post the full response from the above command as it’s lengthy but scanning through it I noticed that the condition of each load balancer pool member was disabled. In the immortal words of Bart Simpson:




The response above is from a more targeted API call to /pools/pool-1.

Now I dont know how it got into this state – maybe someone was doing some jenga style doomsday testing, pulling one brick at a time until the tower crashes! – but this certainly looked to be the cause of the issue. So I figured the quickest fix would be to do a PUT API call to NSX with condition enabled for the pool members and I’d be all set. Not so easy!

Running the following PUT appears to work temporarily (running a GET at the same time confirms this)

But the change does not get fully applied and reverts the conditions to disabled after about 30 seconds with the below error:

So to apply the change to the load balancer NSX requires a handoff with the PSC that is is mapped to…in this case its the load balanced PSC that is not functional. So the command fails.

So it was clear I needed to get at least 1 PSC operational before i could attempt to make a change. Time to play with some DNS redirects to “fool” the PSC services into starting.

As my PSCs are setup in HA mode behind a load balancer the SSO endpoint URL is https://sfo01psc01.sfo01.rainpole.local which both PSCs will respond from. So to get my first PSC up I changed the IP for sfo01psc01.sfo01.rainpole.local in DNS to point to the first PSC’s IP.

So now, pings to the load balancer VIP FQDN sfo01psc01.sfo01.rainpole.local respond from the first PSC IP

Next I set a static entry in /etc/hosts on each of my PSCs, and vCenters to do the same as i’ve seen vCenter especially cache DNS entries in it’s local dnsmasq.

Next step was to stop & start all services on each PSC

service-control –stop –all

service-control –start –all

And hey presto the services started! Ran the same on vCenter and the services also started. This allowed me to go in and modify the load balancer pools to set the members to enabled.

Once the load balancer was back as it should be it was just a case of removing the /etc/hosts entries on each VM and reverting the DNS server change to point the load balancer FQDN back to its correct IP address.

For completeness I restarted all the services on each appliances in the above mentioned order

Moral of the story? Dont disable both nodes in a load balancer pool!

Now onwards with the original testing i needed to do!

Setup multiple vRA tenants using powershell and the vRA7 REST API

Following on from my post on Creating a local user in vRA7 using the REST API i wanted to try and script the entire process of creating multiple vRealize Automation 7 tenants as in our lab as we often need to spin up multiple tenants for testing or development purposes.

Some assumptions:

  • Each tenant has the same prefix of “dev-” and is appended with a 3 digit number starting at “001”
  • Each tenant gets the same local user created with matching credentials
  • Each tenant gets the same AD directory added
  • Each tenant gets the same AD groups added

This script will do the following:

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So as to avoid the requirement to edit the powershell script directly i put all configuration variables in an external .cfg file. This file needs to be placed in the same directory as the powershell script.

Firstly here is the config file contents. Edit each variable to match your environment. Modify the numberOfTenants variable to set the number of tenants you want to create. The example below will create 20 tenants.

[vRA FQDN]
VRA=vra-vip.domain.local

[vRA Credentials to acquire authentication token]
vRAUsername=administrator@vsphere.local
vRAPassword=Password123!
vRADefaultTenant=vsphere.local

[Create tenant details]
numberOfTenants=2
tenantIDPrefix=dev-
tenantURLPrefix=dev-
tenantNamePrefix=dev-
tenantDescription=DevelopmentTenant
tenantemailAddress=admin@vsphere.local

[Local Admin User Details]
firstName=vRA
lastName=Admin
emailAddress=vraadmin@vsphere.local
description=vRAAdmin
locked=false
disabled=false
password=Password123!
domain=vsphere.local
userName=vraadmin
name=vraadmin

[Tenant Directory Details]
adDomain=domain.local
adDomainalias=Domain
type=AD
adUserNameDn=cn=adbind_vra,OU=EHC,DC=domain,DC=local
adBindPassword=Password123!
adURL=ldap://domain.local:389
adGroupBaseSearchDn=ou=EHC,DC=domain,DC=local
adUserBaseSearchDn=ou=EHC,DC=domain,DC=local

[AD Domain Groups to add as Tenant & IaaS Admins]
tenantAdmins=EHC_Tenant_Admins@domain.local
tenantRoleID=CSP_TENANT_ADMIN
iaasAdmins=EHC_IaaS_Admins@domain.local
iaasRoleID=COM_VMWARE_IAAS_IAAS_ADMINISTRATOR

And here is the script to create the tenants. It is broken up into multiple functions

# Script to create vRA7 Tenants in bulk
# Ensure you update the associated cfg file
# with the details of your vRA environment
# and details of the tenants you wish to create
# Created by Brian O'Connell
# Version 1.0.0

# Import configuration variables from external cfg file
Get-Content createvRATenants.cfg | Foreach-Object{
if ($_.length -gt 0) {
 $var = $_ -Split '=',2
 New-Variable -Name $var[0] -Value $var[1]
 }
 } 

Function getvRAAuthToken {
# Construct credentials from config file
$credentials=@{username=$vRAUsername;password=$vRAPassword;tenant=$vRADefaultTenant}
############# Get Auth token ###############
$headers=@{
 "Accept"="application/json"
}
$Global:token = Invoke-RestMethod -Uri "https://$($VRA)/identity/api/tokens" -Method Post -Headers $headers -ContentType application/json -Body (ConvertTo-Json $credentials) | Select -ExpandProperty id
Write-Host "vRA Authentication Token Acquired" -ForegroundColor Green
 } 

Function createvRATenant {
 # ############ Create Tenant ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Create the Tenant
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
$tenantid = -join ($tenantIDPrefix,$tenantNumber)
$tenantURL = -join ($tenantURLPrefix,$tenantNumber)
$tenantName = -join ($tenantNamePrefix,$tenantNumber)
$tenantBody= @"
{
 "@type": "Tenant",
 "id": "$tenantid",
 "urlName": "$tenantURL",
 "name": "$tenantName",
 "description": "$tenantDescription",
 "contactEmail": "$tenantemailAddress"
}
"@ 

$createTenant = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/tenants/$($tenantID)" -headers $headers -ContentType application/json -body $tenantBody
Write-Host "Tenant $($tenantName) created successfully" -ForegroundColor Green
}
 }

Function createvRALocalAdminUser {
 ############# Create Local Admin User ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$userBody= @"
{ "@type": "User",
 "firstName": "$firstName",
 "lastName": "$lastName",
 "emailAddress": "$emailAddress",
 "description": "$description",
 "locked": false,
 "disabled": false,
 "password": "$password",
 "domain": "$domain",
 "userName": "$userName",
 "principalId": {
 "domain": "$domain",
 "name": "$name"
 }
}
"@

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the user
$createUser = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/principals" -headers $headers -ContentType "application/json" -body $userBody
Write-Host "Local Admin User for tenant $($tenantid) created successfully" -ForegroundColor Green
} 

 }

Function updatevRALocalAdminUserRoles {
 ############# Add Local Admin User to Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$principal = "vraadmin@vsphere.local"
$roleIDs = @("CSP_TENANT_ADMIN","COM_VMWARE_IAAS_IAAS_ADMINISTRATOR")

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Add the user to tenant & IaaS admins
foreach ($roleID in $roleIDs) {
$makeUserAdmin = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($principal)/roles/$($roleID)" -headers $headers -body "{}"
}
Write-Host "Local Admin User Added to Tenant & IaaS Admins for tenant $($tenantid) " -ForegroundColor Green
 }
 }

Function createvRATenantDirectory {
 ############# Add AD Tenant directory ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

$directoryBody= @"
{"@type": "IdentityStore",
"domain": "$adDomain",
"name": "$adDomain",
"alias": "$adDomainalias",
"type": "$type",
"userNameDn": "$adUserNameDn",
"password": "$adBindPassword",
"url": "$adURL",
"groupBaseSearchDn": "$adGroupBaseSearchDn",
"userBaseSearchDn": "$adUserBaseSearchDn"
}
"@
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the directory
$createDirectory = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/directories" -headers $headers -ContentType "application/json" -body $directoryBody
Write-Host "Tenant Directory Created for tenant $($tenantid) " -ForegroundColor Green
}

 }

Function addDomainGroupstovRAAdmins {
############## Add AD Domain Groups to vRA Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Add the user to tenant & IaaS admins
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
$addTenantAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($tenantAdmins)/roles/$($tenantRoleID)" -headers $headers -body "{}"

$addIaaSAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($iaasAdmins)/roles/$($iaasRoleID)" -headers $headers -body "{}"
Write-Host "Domain groups added to as tenant & IaaS admins for tenant $($tenantid) " -ForegroundColor Green
}
 }

# Call All functions to setup tenants
getvRAAuthToken; createvRATenant; createvRALocalAdminUser; updatevRALocalAdminUserRoles; createvRATenantDirectory; addDomainGroupstovRAAdmins 

Distributed vRA validation script

From time to time a distributed vRA deployment can have issues…here is a quick script to verify and validate the important components are up and functioning…without the need to log into multiple components. Here is a diagram of my distributed vRA setup

Distributed vRA v4

And here is the script! It has multiple functions to do the following

  • Basic ping tests to each component
  • Get the status of all vRA component services
  • Test the Web & manager server URLs

# Script to check the status of each component of a distributed vRA deployment
# Modify the hostnames to match your environment
$vRAAppliance01FQDN = "vra01.domain.local"
$vRAAppliance02FQDN = "vra02.domain.local"
$vRAWeb01FQDN = "web01.domain.local"
$vRAWeb02FQDN = "web02.domain.local"
$vRAManager01FQDN = "manager01.domain.local"
$vRAManager02FQDN = "manager02.domain.local"
$vRADEM01FQDN = "demw01.domain.local"
$vRADEM02FQDN = "demw02.domain.local"
$vRAAgent01FQDN = "agent01.domain.local"
$vRAAgent02FQDN = "agent02.domain.local"
$vRAComponentServiceURL = "https://vra-vip.domain.local/component-registry/services/status/current"
$webVIPURL = "https://web-vip.domain.local/WAPI"
$managerVIPURL = "https://manager-vip.domain.local/VMPS2"



### DO NOT MODIFY ANYTHING BELOW THIS LINE ###



Function pingHosts {
@"
===============================================================================
Performing basic ping test to each defined component
===============================================================================
"@
$vms = @($vRAAppliance01FQDN, $vRAAppliance02FQDN, $vRAWeb01FQDN, $vRAWeb02FQDN, $vRAManager01FQDN, $vRAManager02FQDN, $vRADEM01FQDN, $vRADEM02FQDN, $vRAAgent01FQDN, $vRAAgent02FQDN)
$collection = $()
foreach ($vm in $vms)
{
 $status = @{ "ServerName" = $vm; "TimeStamp" = (Get-Date -f s) }
 if (Test-Connection $vm -Count 1 -ea 0 -Quiet)
 { 
 $status["Ping Results"] = "Up"
 } 
 else 
 { 
 $status["Ping Results"] = "Down" 
 }
 New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
 $collection += $serverStatus

}
 }
 
Function testvRAServices {
@"
===============================================================================
Getting Status of all vRA component services
===============================================================================
"@
Write-Host "Checking status of vRA Component Services" -ForegroundColor Yellow
# Request Service Information from $vRAComponentServiceURL
$vRAURL = Invoke-WebRequest $vRAComponentServiceURL
# Convert the Json response
$json = ConvertFrom-Json -InputObject $vRAURL.content
# Get Service name & status
$serviceInfo = $json.content
# Loop through each service
foreach ($service in $serviceInfo) {
# Get the Service Name
$serviceName = $service.serviceName
# Get the Service status
$serviceStatus = $service.serviceStatus.serviceInitializationStatus
# If Service Status is blank report it as BLANK POSSIBLY STOPPED
 if (!$serviceStatus) {
 $serviceStatus = "BLANK - POSSIBLY STOPPED?"
 }
# If Service Status is FAILED print to screen in red 
if ($serviceStatus -eq "FAILED") {
 write-host "$serviceName $serviceStatus" -ForeGroundColor Red
 }
# Otherwise print to screen as normal (Remove this if you only want to report failed services) 
 else {
 Write-Host "$serviceName $serviceStatus"}
 }

}


Function testWebVIP {
@"
===============================================================================
Checking status of vRA Web API URL
===============================================================================
"@
Write-Host "Testing IaaS Web Service VIP URL $webVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($webVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Web Service is OK!" -ForegroundColor Green
 # Close HTTP request
$HTTP_Response.Close()
}
Else {
 Write-Host "IaaS Web Service is not responding. Restart IIS on Web01. If that does not resolve then Reboot Web01" -ForegroundColor Red
}
 }
 
Function testMgrVIP {
@"
===============================================================================
Checking status of vRA Manager API URL
===============================================================================
"@
Write-Host "Testing IaaS Manager Service VIP URL $managerVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($managerVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Manager Service is OK!" -ForegroundColor Green
 # Close HTTP request
$HTTP_Response.Close()
}
Else {
 Write-Host "IaaS Manager Service is not responding. Ensure all vRA services are running on manager01. If that does not resolve then Reboot manager01" -ForegroundColor Red
}
 } 
 

 pingHosts; testvRAServices; testWebVIP; testMgrVIP

The function pingHosts is a basic ping test to each defined vRA component

The function testvRAServices was an interesting one to write as I’m not overly familiar with working with APIs so it was a learning experience. I wanted to be able to report the status of all vRA services listed on the VAMI administration UI of a vRA appliance (https://vra:5480). The URL that the services are listed on is https://vra-vip.domain.local/component-registry/services/status/current so using the powershell Invoke-WebRequest you get back the page information.

Invoke-WebRequest

Line 56 in the script puts the page contents into a variable we can work with. You can see that the information we want is stored in Content (ServiceStatus) in Json format so you need to take that Json and convert it to  to powershell readable text using the ConvertFrom-Json function (ConvertFrom-Json converts a JSON-formatted string to a custom object (PSCustomObject) that has a property for each field in the JSON string) Line 58 does this

We then put each service into the $serviceinfo variable and loop through them to get the service name and service status.

Side note here: Originally I was querying $json.content.serviceStatus to get the details i wanted but i noticed I wasnt getting a full list of services, i was getting some blank content and also some duplicate service names. This is how i was doing it

$vRAURL = Invoke-WebRequest "https://vra-vip.vlab.local/component-registry/services/status/current" 
$json = ConvertFrom-Json -InputObject $vRAURL.content 
$serviceInfo = $json.content.serviceStatus | Select serviceName,serviceInitializationStatus $serviceInfo 

Here is that that returns. As you can see its not the full list and there is a duplicate entry so its not much use

Duplicate Results

I dug a little into the API and it seems that it does indeed contain inconsistent information. Here is an excerpt with some issues where the actual service name is content-management but the serviceStatus reports the name as advanced-designer-service

Service Name issue

So to get an accurate list i queried the serviceName field to get the name and the serviceStatus.serviceInitializationStatus to get the service status. Unfortunately doing it this way doesnt allow creating a nice formatted table (at least i havent figured out how to do it yet!) but i did get it to print out each service & status on the same line.

Line 68: In my lab i use a vRO appliance so the internal vRO service on the vRA appliance is stopped. The service status comes back blank so i added a check to report blank service status as “BLANK – POSSIBLY STOPPED?”.

Line 72: I also added a check to print any failed services in red so they stand out.

The testWebVIP and testManagerVIP functions use the powershell System.Net.WebRequest to get the HTTP status code for a given URL. If the status code comes back as 200 then everything is ok. If not there is an issue with your IaaS components

So there you have it. A quick way to verify the status of all of the important vRealize Automation components and services. In my example below the iaas-service is in a failed state (The driving reason for creating this script! 🙂 )

Script Results

Create a local Tenant user in vRA7 using the rest API

VMware vRealize Automation 7.x is the latest version of VMware’s cloud automation software. With the new version comes a change in the way tenants (and user roles for same) are setup. In previous versions you would do the following to create a tenant

  • Log into the default tenant
  • Create a new tenant
  • Add an identity store
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

With vRA 7.x the procedure changes (and becomes more cumbersome IMO)

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So when it comes to trying to automate tenant creation its not as easy as it used to be for 6.x. For 6.x I’ve used cloud client (if you’re not familiar with CloudClient i’d recommend checking it out here https://developercenter.vmware.com/tool/cloudclient/4.1.0 )

The CloudClient commands required for 6.x are as follows:

  • Firstly login to CloudClient
vra login userpass --server vra-vip.domain.local --user administrator@vsphere.local --password Password123! --tenant vsphere.local
  • Next create the tenant
vra tenant add --name NewTenant --url newtenant

  • Next add the identity store for AD authentication
vra tenant identitystore add --tenantname NewTenant --name Domain.local --url ldap://domain.local --groupbasedn 'ou=vRA,DC=domain,DC=local' --domain domain.local --userdn 'cn=adbind_vra,OU=vRA,DC=domain,DC=local' --password Password123! --type AD --userbasedn 'ou=vRA,DC=domain,DC=local'

  • Next add Tenant administrators to the new tenant
vra tenant admin update --tenantname NewTenant --role TENANT_ADMIN --action ADD --users vRA_Tenant_Admins@domain.local

  • Next add IaaS Administrators to the new tenant
vra tenant admin update --tenantname NewTenant --role IAAS_ADMIN --action ADD --users vRA_IaaS_Admins@domain.local

 

So enter vRA 7.x. Using CloudClient the first 2 steps are the same as before

  • Firstly login to CloudClient
vra login userpass --server vra-vip.domain.local --user administrator@vsphere.local --password Password123! --tenant vsphere.local

  • Next create the tenant
vra tenant add --name NewTenant --url newtenant

  • Next we need to create a local user in the tenant. I couldn’t find a function in CloudClient to create the local user. I checked the developer API guide and I also tried the excellent community module PowervRA http://www.jonathanmedd.net/2016/03/introducing_powervra.html but no joy so i enlisted the help of my colleague Sean Leahy @leahy_s to query the REST API. We ran the add user operation manually and monitored the process in FireFox (Press F12 to get access to the browser console debugger.) Manually enter the user details and click Ok and you will see a POST operation in the console. See below

 

F12 Console

  • Click on the POST operation and inspect the REST information. In the headers tab you can see the Content-Type is application/json and in the params tab you can see the post data

F12 Params

  • Using the API guide we found the required REST method to get an authentication token from vRA that will allow us to POST data. In this example i’m using Advanced REST client in Chrome to retrieve the auth token from vRA. So the URL to POST to is https://vra-appliance-FQDN/identity/api/tokens and the credentials payload needs to be JSON formatted. See below

RESt Auth to vRA

  • The response back should contain a base64 encoded string that will be used as an authorization token to post the new user.
  • So again using Advanced REST client in chrome (or your preferred method) we need to do a POST operation to create the user. This time you need to use URL https://vra-appliance-FQDN/identity/api/tenants/NewTenant/principals. In the headers you need Accept set to Application/json and Authorization set to the base64 encoded token string we got in the previous step. IMPORTANT: You must prefix the token with Bearer or it will not work! See below

 

POST User

  • Here is the full JSON payload as you cant see it all in the screenshot
{"@type": "User",
 "firstName": "vRA",
 "lastName": "Admin",
 "emailAddress": "vraadmin@domain.local",
 "description": "vRA Admin",
 "locked": false,
 "disabled": false,
 "password": "Password123!",
"domain": "vsphere.local",
"userName":"vraadmin",
 "principalId": {"domain": "vsphere.local",
 "name": "vraadmin"}
 }
  • Ok deep breaths…if you’re still with me you can now proceed to adding the user to the tenant and IaaS administrator groups!
  • In the interest of trying out different tools I decided to try using PowervRA for this task. (I will revisit this using the REST API directly next week) I wont go into installing PowervRA. There are good instructions here https://github.com/jakkulabs/PowervRA
  • In PowervRA run the following commands
Add-vRAPrincipalToTenantRole -TenantId NewTenant -PrincipalId vraadmin@vsphere.local -RoleId CSP_TENANT_ADMIN

Add-vRAPrincipalToTenantRole -TenantId NewTenant -PrincipalId vraadmin@vsphere.local -RoleId COM_VMWARE_IAAS_IAAS_ADMINISTRATOR

Not sure why the complete difference in RoleId name format but thats the only way i could get it to work!

So now we have a tenant with a local user that is both tenant admin & IaaS admin but we still dont have an AD directory service to authenticate AD users. So for this we will again leverage the REST API. Using the same auth token as before, this time we need to hit https://vra-appliance-FQDN/identity/api/tenants/NewTenant/directories

POST Identity Store

  • Here is the full JSON payload as you cant see it all in the screenshot
{
@type: "IdentityStore",
domain: "domain.local",
name: "Domain.local",
alias: "Domain",
type: "AD",
userNameDn: "cn=adbind_vra,OU=EHC,DC=domain,DC=local",
password: "Password123!",
url: "ldap://domain.local:389",
groupBaseSearchDn: "ou=EHC,DC=domain,DC=local",
userBaseSearchDn: "ou=EHC,DC=domain,DC=local"
}

  • We can now go and add our domain users as Tenant admins and IaaS admins to allow domain users to log into the new tenant! So its a longer process than 6.x but its still doable (even if parts of it are undocumented in the API guide!). I will be working on scripting this for multiple tenants so hopefully keep an eye out for a follow up post as i fumble my way through using the vRealize Automation REST API! 🙂

Special thanks again to my colleague Sean Leahy for the REST pointers!