Site Protection & Disaster Recovery for VMware Cloud Foundation Validated Solution

Along with the release of VMware Cloud Foundation 4.3.1, we are excited to announce the general availability of the Site Protection & Disaster Recovery for VMware Cloud Foundation Validated Solution. The solution documentation, intro and other associated collateral can be found on the Cloud Platform Tech Zone here.

The move from VMware Validated Designs to VMware Validated Solutions has been covered by my team mate Gary Blake in detail here so I wont go into that detail here. Instead I will concentrate on the work Ken Gould and I (along with a supporting team) have been working to deliver for the past few months.

The Site Protection & Disaster Recovery for VMware Cloud Foundation Validated Solution includes the following to deliver an end-to-end validated way to protect your mission critical applications. You get a set of documentation that is tailored to the solution that includes: design objectives, a detailed design including not just design decisions, but the justifications & implications of those decisions, detailed implementation steps with PowerShell alternatives for some steps to speed up time to deploy, operational guidance on how to use the solution once its deployed, solution interoperability between it and other Validated Solutions, an appendix containing all the solution design decisions in one easy place for review, and finally, a set of frequently asked questions that will be updated for each release.

Disaster recovery is a huge topic for everyone lately. Everything from power outages to natural disasters to ransomware and beyond can be classed as a disaster, and regardless of the type of disaster you must be prepared. To adequately plan for business continuity in the event of a disaster you must protect your mission critical applications so that they may be recovered. In a VMware Cloud Foundation environment, cloud operations and automation services are delivered by vRealize Lifecycle Manager, vRealize Operations Manager & vRealize Automation, with authentication services delivered by Workspace ONE Access.

To provide DR for our mission critical apps we leverage 2 VCF instances with NSX-T federation between them. The primary VCF instance runs the active NSX-T global manager and the recovery VCF instance runs the standby NSX-T global manager. All load balancing services are served from the protected instance, with a standby load balancer (disconnected from the recovery site NSX Tier-1 until required, to avoid IP conflicts) in the recovery instance. Using our included PowerShell cmdlets you can quickly create and configure the standby load balancer to mimic your active load balancer, saving you a ton of manual UI clicks.

In the (hopefully never) event of the need to failover the cloud management applications, you can easily bring the standby load balancer online to enable networking services for the failed over applications.

Using Site recovery Manager (SRM) you can run planned migrations or disaster recovery migrations. With a single set of SRM recovery plans, regardless of the scenario, you will be guided through the recovery process. In this post I will cover what happens in the event of a disaster.

When a disaster occurs on the protected site (once the panic subsides) there are a series of tasks you need to perform to bring those mission critical apps back online.

First? Fix the network! Log into the passive NSX Global Manager (GM) on the recovery site and promote the GM to Active. (Note: This can take about 10-15 mins)

To cover the case of an accidental “Force Active” click..we’ve built in the “Are you absolutely sure this is what you want to do?” prompt!

Once the promotion operation completes our standby NSX GM is now active, and can be used to manage the surviving site NSX Local Manager (LM)

Once the recovery site GM is active we need to ensure that the cross-instance NSX Tier-1 is now directing the egress traffic via the recovery site. To do this we must update the locations on the Tier-1. Navigate to GM> Tier-1 gateways > Cross Instance Tier-1. Under Locations, make the recovery location Primary.

The next step is to ensure we have an active load balancer running in the recovery site to ensure our protected applications come up correctly. To do this log into what is now our active GM, select the recovery site NSX Local Manager (LM), and navigate to Networking > Load Balancing. Edit the load balancer and attach it to the recovery site standalone Tier-1.

At this point we are ready to run our SRM recovery plans. The recommended order for running the recovery plans (assuming you have all of the protected components listed below) is as follows. This ensures lifecycle & authentication services (vRSLCM & WSA) are up before the applications that depend on them (vROPS & vRA)

  • vRSLCM – WSA – RP
  • Intelligent Operations Management RP
  • Private Cloud Automation RP

I’m not going to go through each recovery plan in detail here. They are documented in the Site Protection and Disaster Recovery Validated Solution. In some you will be prompted to verify this or that along the way to ensure successful failover.

The main thing in a DR situation is, DO NOT PANIC. And what is the best way to getting to a place where you DO NOT PANIC? Test your DR plans…so when you see this…

Your reaction is this…

Trust the plan…test the plan…relax…you have a plan!

Hopefully this post was useful..if you want to learn more please reach out in the comments…if you’re attending VMworld and would like to learn more or ask some questions, please drop into our Meet The Experts session on Thursday.

Take a look at Ken’s post on the Planning & Preparation Workbook for this validated solution for more details.

PowerShell Script to Configure an NSX-T Load Balancer for the vRealize Suite & Workspace ONE Access

As part of my role in the VMware Hyper-converged Business Unit (HCIBU) I spend a lot of time working with new product versions testing integrations for next-gen VMware Validated Designs and Cloud Foundation. A lot of my focus is on Cloud Operations and Automation (vROPs, vRLI, vRA etc) and consequently I regularly need to deploy environments to perform integration testing. I will typically leverage existing automation where possible and tend to create my own when i find gaps. Once such gap was the ability to use PowerShell to interact with the NSX-T API. For anyone who is familiar with setting up a load balancer for the vRealize Suite in NSX-T – there are a lot of manual clicks required. So i set about creating some PowerShell functions to make it a little less tedious and to speed up getting my environments setup so i could get to the testing faster.

There is comprehensive NSX-T API documentation posted on code.vmware .com that I used to decipher the various API endpoints required to complete the various tasks:

  • Create the Load Balancer
  • Create the Service Monitors
  • Create the Application Profiles
  • Create the Server Pools
  • Create the Virtual Servers

The result is a PowerShell module with a function for each of the above and a corresponding JSON file that is read in for the settings for each function. I have included a sample JSON file to get you started. Just substitute your values.

Note: You must have a Tier-1 & associated segments created. (I’ll add that functionality when i get a chance!)

PowerShell Module, Sample JSON & Script are posted to Github here

Cleanup failed requests in vRA UI

From time to time a request in vRA will fail for whatever reason. When this happens you will see the request status as failed on the requests tab. There is a greyed out delete button that for whatever reason cannot be used to delete the failed request even when logged in as a full tenant/iaas/cloud admin.


There are several reasons you may want to remove failed requests…maybe you may need to deliver a demo to the CIO on some new functionality and failures in the UI never look good…or maybe you just have mild OCD like me and like to cleanup any failures to restore the illusion of all being good with the world! ūüôā Whatever your reasons here is a procedure that you can use.

Disclaimer: I dont believe this procedure if fully supported by VMware so please proceed with caution.

  • SSH to your primary vRA appliance
  • Run the following to view the contents of¬†/etc/vcac/server.xml
    • less /etc/vcac/server.xml
  • Look for the line with password= and copy everything between the “”. This password will allow you to connect to the vRA PostGres DB

  • Run the following command with the password from the above step
    • vcac-config prop-util -d –p “s2enc~K6RsAv5WGpoAt+qsnZPrKErxZ0kU1npeK/G5iMzyaWI=”
  • Next change to the postgres user
    • su postgres
  • Change to the postgres directory
    • cd /opt/vmware/vpostgres/current/bin
  • Connect to the vcac database
    • ./psql vcac -W
  • Enter the password from server.xml
  • vRA requests are store in the cat_request table. To enable us to delete a request we first need the request id. Query the cat_request table for your request ID using the requestnumber (In my case the offending failed requestnumber is 63, as seen in the first column in the screenshot above. replace with your requestnumber)
    • SELECT id,requestnumber FROM cat_request where requestnumber = ’63’;

vRA XaaS blueprint requests are referenced in 1 further table, cat_requestevent. This entriy must be deleted before you can delete the request.

  • Run the following commands to delete the request.
  • delete from cat_requestevent where request_id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;
  • delete from cat_request where id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;

The offending failed request should now be gone from the requests list in vRA!

vRA Network and Security Inventory Data Collection Failed

I’ve been playing around with Dell EMC RP4VM & vRA and needed to setup cross vCenter NSX in my lab. I’m not going to go into that setup as there are many blogs on the subject. What i will cover is an error i hit when trying to do¬†Network and Security Inventory data collections on one of my NSX endpoints. The error from the Dem logs in vRA (Infrastructure > Monitoring > Log ) was as follows:

Workflow 'vSphereVCNSInventory' failed with the following exception:
'object' does not contain a definition for 'clusters'

After digging around for VMware KBs and blogs on the subject and coming up empty handed i went back to review my entire setup and discovered i had missed adding a vCenter cluster to the universal transport zone on the offending NSX endpoint, which is my DR site.

Once i rectified this the Network and Security Inventory Data Collection worked as expected.

Setup multiple vRA tenants using powershell and the vRA7 REST API

Following on from my post on Creating a local user in vRA7 using the REST API i wanted to try and script the entire process of creating multiple vRealize Automation 7 tenants as in our lab as we often need to spin up multiple tenants for testing or development purposes.

Some assumptions:

  • Each tenant has the same prefix of “dev-” and is appended with a 3 digit number starting at “001”
  • Each tenant gets the same local user created with matching credentials
  • Each tenant gets the same AD directory added
  • Each tenant gets the same AD groups added

This script will do the following:

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So as to avoid the requirement to edit the powershell script directly i put all configuration variables in an external .cfg file. This file needs to be placed in the same directory as the powershell script.

Firstly here is the config file contents. Edit each variable to match your environment. Modify the numberOfTenants variable to set the number of tenants you want to create. The example below will create 20 tenants.


[vRA Credentials to acquire authentication token]

[Create tenant details]

[Local Admin User Details]

[Tenant Directory Details]

[AD Domain Groups to add as Tenant & IaaS Admins]

And here is the script to create the tenants. It is broken up into multiple functions

# Script to create vRA7 Tenants in bulk
# Ensure you update the associated cfg file
# with the details of your vRA environment
# and details of the tenants you wish to create
# Created by Brian O'Connell
# Version 1.0.0

# Import configuration variables from external cfg file
Get-Content createvRATenants.cfg | Foreach-Object{
if ($_.length -gt 0) {
 $var = $_ -Split '=',2
 New-Variable -Name $var[0] -Value $var[1]

Function getvRAAuthToken {
# Construct credentials from config file
############# Get Auth token ###############
$Global:token = Invoke-RestMethod -Uri "https://$($VRA)/identity/api/tokens" -Method Post -Headers $headers -ContentType application/json -Body (ConvertTo-Json $credentials) | Select -ExpandProperty id
Write-Host "vRA Authentication Token Acquired" -ForegroundColor Green

Function createvRATenant {
 # ############ Create Tenant ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Create the Tenant
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
$tenantid = -join ($tenantIDPrefix,$tenantNumber)
$tenantURL = -join ($tenantURLPrefix,$tenantNumber)
$tenantName = -join ($tenantNamePrefix,$tenantNumber)
$tenantBody= @"
 "@type": "Tenant",
 "id": "$tenantid",
 "urlName": "$tenantURL",
 "name": "$tenantName",
 "description": "$tenantDescription",
 "contactEmail": "$tenantemailAddress"

$createTenant = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/tenants/$($tenantID)" -headers $headers -ContentType application/json -body $tenantBody
Write-Host "Tenant $($tenantName) created successfully" -ForegroundColor Green

Function createvRALocalAdminUser {
 ############# Create Local Admin User ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$userBody= @"
{ "@type": "User",
 "firstName": "$firstName",
 "lastName": "$lastName",
 "emailAddress": "$emailAddress",
 "description": "$description",
 "locked": false,
 "disabled": false,
 "password": "$password",
 "domain": "$domain",
 "userName": "$userName",
 "principalId": {
 "domain": "$domain",
 "name": "$name"

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the user
$createUser = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/principals" -headers $headers -ContentType "application/json" -body $userBody
Write-Host "Local Admin User for tenant $($tenantid) created successfully" -ForegroundColor Green


Function updatevRALocalAdminUserRoles {
 ############# Add Local Admin User to Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$principal = "vraadmin@vsphere.local"

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Add the user to tenant & IaaS admins
foreach ($roleID in $roleIDs) {
$makeUserAdmin = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($principal)/roles/$($roleID)" -headers $headers -body "{}"
Write-Host "Local Admin User Added to Tenant & IaaS Admins for tenant $($tenantid) " -ForegroundColor Green

Function createvRATenantDirectory {
 ############# Add AD Tenant directory ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

$directoryBody= @"
{"@type": "IdentityStore",
"domain": "$adDomain",
"name": "$adDomain",
"alias": "$adDomainalias",
"type": "$type",
"userNameDn": "$adUserNameDn",
"password": "$adBindPassword",
"url": "$adURL",
"groupBaseSearchDn": "$adGroupBaseSearchDn",
"userBaseSearchDn": "$adUserBaseSearchDn"
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the directory
$createDirectory = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/directories" -headers $headers -ContentType "application/json" -body $directoryBody
Write-Host "Tenant Directory Created for tenant $($tenantid) " -ForegroundColor Green


Function addDomainGroupstovRAAdmins {
############## Add AD Domain Groups to vRA Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Add the user to tenant & IaaS admins
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
$addTenantAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($tenantAdmins)/roles/$($tenantRoleID)" -headers $headers -body "{}"

$addIaaSAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($iaasAdmins)/roles/$($iaasRoleID)" -headers $headers -body "{}"
Write-Host "Domain groups added to as tenant & IaaS admins for tenant $($tenantid) " -ForegroundColor Green

# Call All functions to setup tenants
getvRAAuthToken; createvRATenant; createvRALocalAdminUser; updatevRALocalAdminUserRoles; createvRATenantDirectory; addDomainGroupstovRAAdmins 

Distributed vRA validation script

From time to time a distributed vRA deployment can have issues…here is a quick script to verify and validate the important components are up and functioning…without the need to log into multiple components. Here is a diagram of my distributed vRA setup

Distributed vRA v4

And here is the script! It has multiple functions to do the following

  • Basic ping tests to each component
  • Get the status of all vRA component services
  • Test the Web & manager server URLs

# Script to check the status of each component of a distributed vRA deployment
# Modify the hostnames to match your environment
$vRAAppliance01FQDN = "vra01.domain.local"
$vRAAppliance02FQDN = "vra02.domain.local"
$vRAWeb01FQDN = "web01.domain.local"
$vRAWeb02FQDN = "web02.domain.local"
$vRAManager01FQDN = "manager01.domain.local"
$vRAManager02FQDN = "manager02.domain.local"
$vRADEM01FQDN = "demw01.domain.local"
$vRADEM02FQDN = "demw02.domain.local"
$vRAAgent01FQDN = "agent01.domain.local"
$vRAAgent02FQDN = "agent02.domain.local"
$vRAComponentServiceURL = "https://vra-vip.domain.local/component-registry/services/status/current"
$webVIPURL = "https://web-vip.domain.local/WAPI"
$managerVIPURL = "https://manager-vip.domain.local/VMPS2"


Function pingHosts {
Performing basic ping test to each defined component
$vms = @($vRAAppliance01FQDN, $vRAAppliance02FQDN, $vRAWeb01FQDN, $vRAWeb02FQDN, $vRAManager01FQDN, $vRAManager02FQDN, $vRADEM01FQDN, $vRADEM02FQDN, $vRAAgent01FQDN, $vRAAgent02FQDN)
$collection = $()
foreach ($vm in $vms)
 $status = @{ "ServerName" = $vm; "TimeStamp" = (Get-Date -f s) }
 if (Test-Connection $vm -Count 1 -ea 0 -Quiet)
 $status["Ping Results"] = "Up"
 $status["Ping Results"] = "Down" 
 New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
 $collection += $serverStatus

Function testvRAServices {
Getting Status of all vRA component services
Write-Host "Checking status of vRA Component Services" -ForegroundColor Yellow
# Request Service Information from $vRAComponentServiceURL
$vRAURL = Invoke-WebRequest $vRAComponentServiceURL
# Convert the Json response
$json = ConvertFrom-Json -InputObject $vRAURL.content
# Get Service name & status
$serviceInfo = $json.content
# Loop through each service
foreach ($service in $serviceInfo) {
# Get the Service Name
$serviceName = $service.serviceName
# Get the Service status
$serviceStatus = $service.serviceStatus.serviceInitializationStatus
# If Service Status is blank report it as BLANK POSSIBLY STOPPED
 if (!$serviceStatus) {
 $serviceStatus = "BLANK - POSSIBLY STOPPED?"
# If Service Status is FAILED print to screen in red 
if ($serviceStatus -eq "FAILED") {
 write-host "$serviceName $serviceStatus" -ForeGroundColor Red
# Otherwise print to screen as normal (Remove this if you only want to report failed services) 
 else {
 Write-Host "$serviceName $serviceStatus"}


Function testWebVIP {
Checking status of vRA Web API URL
Write-Host "Testing IaaS Web Service VIP URL $webVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($webVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Web Service is OK!" -ForegroundColor Green
 # Close HTTP request
Else {
 Write-Host "IaaS Web Service is not responding. Restart IIS on Web01. If that does not resolve then Reboot Web01" -ForegroundColor Red
Function testMgrVIP {
Checking status of vRA Manager API URL
Write-Host "Testing IaaS Manager Service VIP URL $managerVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($managerVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Manager Service is OK!" -ForegroundColor Green
 # Close HTTP request
Else {
 Write-Host "IaaS Manager Service is not responding. Ensure all vRA services are running on manager01. If that does not resolve then Reboot manager01" -ForegroundColor Red

 pingHosts; testvRAServices; testWebVIP; testMgrVIP

The function pingHosts is a basic ping test to each defined vRA component

The function testvRAServices was an interesting one to write as I’m not overly familiar with working with APIs so it was a learning experience. I wanted to be able to report the status of all vRA services listed on the VAMI administration UI of a vRA appliance (https://vra:5480). The URL that the services are listed on is https://vra-vip.domain.local/component-registry/services/status/current so using the powershell Invoke-WebRequest you get back the page information.


Line 56 in the script puts the page contents into a variable we can work with. You can see that the information we want is stored in Content (ServiceStatus) in Json format so you need to take that Json and convert it to  to powershell readable text using the ConvertFrom-Json function (ConvertFrom-Json converts a JSON-formatted string to a custom object (PSCustomObject) that has a property for each field in the JSON string) Line 58 does this

We then put each service into the $serviceinfo variable and loop through them to get the service name and service status.

Side note here: Originally I was querying $json.content.serviceStatus to get the details i wanted but i noticed I wasnt getting a full list of services, i was getting some blank content and also some duplicate service names. This is how i was doing it

$vRAURL = Invoke-WebRequest "https://vra-vip.vlab.local/component-registry/services/status/current" 
$json = ConvertFrom-Json -InputObject $vRAURL.content 
$serviceInfo = $json.content.serviceStatus | Select serviceName,serviceInitializationStatus $serviceInfo 

Here is that that returns. As you can see its not the full list and there is a duplicate entry so its not much use

Duplicate Results

I dug a little into the API and it seems that it does indeed contain inconsistent information. Here is an excerpt with some issues where the actual service name is content-management but the serviceStatus reports the name as advanced-designer-service

Service Name issue

So to get an accurate list i queried the serviceName field to get the name and the serviceStatus.serviceInitializationStatus to get the service status. Unfortunately doing it this way doesnt allow creating a nice formatted table (at least i havent figured out how to do it yet!) but i did get it to print out each service & status on the same line.

Line 68: In my lab i use a vRO appliance so the internal vRO service on the vRA appliance is stopped. The service status comes back blank so i added a check to report blank service status as “BLANK – POSSIBLY STOPPED?”.

Line 72: I also added a check to print any failed services in red so they stand out.

The testWebVIP and testManagerVIP functions use the powershell System.Net.WebRequest to get the HTTP status code for a given URL. If the status code comes back as 200 then everything is ok. If not there is an issue with your IaaS components

So there you have it. A quick way to verify the status of all of the important vRealize Automation components and services. In my example below the iaas-service is in a failed state (The driving reason for creating this script! ūüôā )

Script Results

Create a local Tenant user in vRA7 using the rest API

VMware vRealize Automation 7.x is the latest version of VMware’s cloud automation software. With the new version comes a change in the way tenants (and user roles for same) are setup. In previous versions you would do the following to create a tenant

  • Log into the default tenant
  • Create a new tenant
  • Add an identity store
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

With vRA 7.x the procedure changes (and becomes more cumbersome IMO)

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So when it comes to trying to automate tenant creation its not as easy as it used to be for 6.x. For 6.x I’ve used cloud client (if you’re not familiar with CloudClient i’d recommend checking it out here¬†¬†)

The CloudClient commands required for 6.x are as follows:

  • Firstly login to CloudClient
vra login userpass --server vra-vip.domain.local --user administrator@vsphere.local --password Password123! --tenant vsphere.local
  • Next create the tenant
vra tenant add --name NewTenant --url newtenant

  • Next add the identity store for AD authentication
vra tenant identitystore add --tenantname NewTenant --name Domain.local --url ldap://domain.local --groupbasedn 'ou=vRA,DC=domain,DC=local' --domain domain.local --userdn 'cn=adbind_vra,OU=vRA,DC=domain,DC=local' --password Password123! --type AD --userbasedn 'ou=vRA,DC=domain,DC=local'

  • Next add Tenant administrators to the new tenant
vra tenant admin update --tenantname NewTenant --role TENANT_ADMIN --action ADD --users vRA_Tenant_Admins@domain.local

  • Next add IaaS Administrators to the new tenant
vra tenant admin update --tenantname NewTenant --role IAAS_ADMIN --action ADD --users vRA_IaaS_Admins@domain.local


So enter vRA 7.x. Using CloudClient the first 2 steps are the same as before

  • Firstly login to CloudClient
vra login userpass --server vra-vip.domain.local --user administrator@vsphere.local --password Password123! --tenant vsphere.local

  • Next create the tenant
vra tenant add --name NewTenant --url newtenant

  • Next we need to create a local user in the tenant. I couldn’t find a function in CloudClient to create the local user. I checked the developer API guide and I also tried the excellent community module PowervRA¬†¬†but no joy so i enlisted the help of my colleague Sean Leahy¬†@leahy_s¬†to query the REST API. We ran the add user operation manually and monitored the process in FireFox (Press F12 to get access to the browser console debugger.) Manually enter the user details and click Ok and you will see a POST operation in the console. See below


F12 Console

  • Click on the POST operation and inspect the REST information. In the headers tab you can see the Content-Type is application/json and in the params¬†tab you can see the post data

F12 Params

  • Using the API guide we found the required REST method¬†to get an authentication token from vRA that will allow us to POST data. In this example i’m using Advanced REST client in Chrome to retrieve the auth token from vRA. So the URL to POST to is https://vra-appliance-FQDN/identity/api/tokens and the credentials payload needs to be JSON formatted. See below

RESt Auth to vRA

  • The response back should contain a base64 encoded string that will be used as an authorization token to post the new user.
  • So again using Advanced REST client in chrome (or your preferred method) we need to do a POST operation to create the user. This time you need to use URL https://vra-appliance-FQDN/identity/api/tenants/NewTenant/principals. In the headers you need Accept set to Application/json and Authorization set to the base64 encoded token string we got in the previous step. IMPORTANT: You must prefix the token with Bearer or it will not work! See below



  • Here is the full JSON payload as you cant see it all in the screenshot
{"@type": "User",
 "firstName": "vRA",
 "lastName": "Admin",
 "emailAddress": "vraadmin@domain.local",
 "description": "vRA Admin",
 "locked": false,
 "disabled": false,
 "password": "Password123!",
"domain": "vsphere.local",
 "principalId": {"domain": "vsphere.local",
 "name": "vraadmin"}
  • Ok deep breaths…if you’re still with me you can now¬†proceed to adding the user to the tenant and IaaS administrator groups!
  • In the interest of trying out different tools I decided to try using PowervRA for this task. (I will revisit this using the REST API directly next week) I wont go into installing PowervRA. There are good instructions here¬†
  • In PowervRA run the following commands
Add-vRAPrincipalToTenantRole -TenantId NewTenant -PrincipalId vraadmin@vsphere.local -RoleId CSP_TENANT_ADMIN

Add-vRAPrincipalToTenantRole -TenantId NewTenant -PrincipalId vraadmin@vsphere.local -RoleId COM_VMWARE_IAAS_IAAS_ADMINISTRATOR

Not sure why the complete difference in RoleId name format but thats the only way i could get it to work!

So now we have a tenant with a local user that is both tenant admin & IaaS admin but we still dont have an AD directory service to authenticate AD users. So for this we will again leverage the REST API. Using the same auth token as before, this time we need to hit https://vra-appliance-FQDN/identity/api/tenants/NewTenant/directories

POST Identity Store

  • Here is the full JSON payload as you cant see it all in the screenshot
@type: "IdentityStore",
domain: "domain.local",
name: "Domain.local",
alias: "Domain",
type: "AD",
userNameDn: "cn=adbind_vra,OU=EHC,DC=domain,DC=local",
password: "Password123!",
url: "ldap://domain.local:389",
groupBaseSearchDn: "ou=EHC,DC=domain,DC=local",
userBaseSearchDn: "ou=EHC,DC=domain,DC=local"

  • We can now go and add our domain users as Tenant admins and IaaS admins to allow domain users to log into the new tenant! So its a longer process than 6.x but its still doable (even if parts of it are undocumented in the API guide!). I will be working on scripting this for multiple tenants so hopefully keep an eye out for a follow up post as i fumble my way through using the vRealize Automation REST API! ūüôā

Special thanks again to my colleague Sean Leahy for the REST pointers!

Failed vRA IaaS Web Server Install

There are many reasons why a vRealize Automation IaaS Web server install can fail

  • MSDTC Issues
  • NTP Issues
  • DNS Issues
  • Certificate Issues
  • Mental Issues…. (Caused by all of the above!)

I hit this error in the lab and the fix was a new one to me so said i’d document it. The IaaS web server was failing to install and the error in the logs is below which pointed to the certs

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

I checked and double checked all certificates and SAN attributes of each component, checked DNS resolution forward and reverse, verified NTP etc. Then it occurred to me that the signing CA is from a different domain than the domain i was installing in so i added the root cert to the local trusted root certification store and low and behold the install completed successfully. Had to do the same on all other IaaS VMs. So even though the root and signing certs were in the personal store with the SAN certs it was not sufficient.


vRA Infrastructure tab 404 error

A 404 error when accessing the infrastructure tab can be cause by a number of things. Expired certs is the most common reason. I hit this issue earlier but i’m using valid CA certs. When i logged into the VAMI¬†interface for vRA i noticed that the IaaS service was not registered. After the customary reboots the issue persisted so i had to did a little deeper. Here is what i saw in VAMI/Services.(Not using the embedded vCO so it is intentionally disabled)

VAMI Services

To get the IaaS service registered again i had to re register IaaS with vRA using the following commands on the IaaS server


  • cd c:\‚ÄĚProgram Files (x86)‚ÄĚ\VMware\vCAC\Server\‚ÄĚModel Manager Data‚ÄĚ\Cafe

vRA 6.2 Distributed IaaS deployment

I needed to deploy a distributed vRA 6.2 IaaS in the lab and didn’t find too many resources on the web so here are some of my notes! I hope to revisit this soon to do a full distributed HA deployment including NSX load balancers. For now here is a standard distributed deployment


  • AD installed & available
  • vCenter/SSO installed & available
  • SQL Server installed and available and IaaS SQL pre reqs run for MSDTC
  • vRA 6.2 appliance deployed and integrated with SSO
  • AD CA configured (Will try to cover this in another post)
  • 5 Windows 2012 R2 VMs. 1 for each of the following roles – Web Server, Manager Server, DEM Orchestrator, DEM Worker, IaaS Agent
  • All windows VMs joined the AD domain
  • Run Brian Graf’s¬†IaaS pre req script¬†on all VMs (Pretty sure its only required on the Web & manager server but didnt take any chances!)

Here are the steps

  • Download the IaaS installer from the vRA appliance @ https://vra-FQDN:5480/i
  • First we need to install the database component. From any VM run the IaaS installer
  • Accept the ELUA and click Next
  • Enter the root credentials for the vRA appliance and select Accept Certificate and click Next
  • Choose Custom Install > IaaS Server

Installation Type

  • Select Database from the¬†features list and enter the database instance and database name and click Next (I’m using SQL Express in the lab)

2.install DB

  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

3.DB Pre Reqs

  • Click Install on the Ready to Install Screen and Next and Finish once the Database install completes
  • Next you need to install the Website feature
  • Before installing the website component it is recommended to generate a CA signed cert for the server FQDN.
  • To do this open IIS on the Web server, select your server name and select Server Certificates
  • In the right hand pane select Create Domain Certificate

Generate Certificate

  • Enter the certificate distinguished name for the web server and click Next

Certificate Request


  • Click Select to choose your CA and assign a friendly name (user the web server FQDN)
  • Once complete¬†run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Website from the feature list.
  • Choose the Default Web Site and the default port of 443
  • Under Available certificates choose the CA signed certificate you created earlier and enter the web server FQDN for the IaaS Server

web server setup

  • Because the IaaS Model Manager Data is required for the Website component to run you must also select the ModelManagerData feature
  • Enter the vCAC/vRA appliance FQDN & SSO details and also the IaaS Web Server FQDN and click Next

9. ModelManagerData


  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

10. ModelManagerPreReqs

  • On the Server and Account Settings screen enter the password for the service account being used and a passphrase for the database and click Next and click Finish once the install completes

11. ModelManagerData Account settings

  • Next up is the Manager Service
  • On the VM you designated for the¬†Manager Service follow the steps outlined earlier to generate a CA signed certificate
  • Once complete¬†run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Manager Service¬†from the feature list.
  • Enter the FQDN of the Web Server and select the certificate you created in the previous steps and click Next and click Finish

managerService Install


  • Next up is the DEM Orchestrator
  • On the VM you designated for the DEMO Orchestrator run the IaaS installer, choose custom > Distributed Execution Managers and click Next
  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Distributed Execution Manager Screen do the following
    • From the DEM role drop down choose Orchestrator
    • Enter a name
    • Enter a description
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Click Add and click Next and click Install and then click Finish

DEM Orchestratot Config


  • Repeat the above steps on the DEM Worker VM choosing the DEM Worker Role
  • Finally we will install a Proxy Agent that will be used to communicate with vRA endpoints
  • On the VM you designated for the IaaS Agent¬†run the IaaS installer, choose custom >¬†Proxy Agents and click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Proxy Agent¬†Screen do the following
    • From the Agent Type¬†drop down choose¬†vSphere
    • Enter a name
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Enter the vCenter Endpoint FQDN
    • Click Add and click Next and click Install and then click Finish

Proxy Agent Install

  • If everything went according to plan you should now be able to log into vRA and configure tenants and resources and start using your distributed IaaS installation.

Hopefully if you’ve read this far you found this useful. I hope to post the distributed HA procedure soon once i get some free lab time!