Cleanup failed requests in vRA UI

From time to time a request in vRA will fail for whatever reason. When this happens you will see the request status as failed on the requests tab. There is a greyed out delete button that for whatever reason cannot be used to delete the failed request even when logged in as a full tenant/iaas/cloud admin.


There are several reasons you may want to remove failed requests…maybe you may need to deliver a demo to the CIO on some new functionality and failures in the UI never look good…or maybe you just have mild OCD like me and like to cleanup any failures to restore the illusion of all being good with the world! ūüôā Whatever your reasons here is a procedure that you can use.

Disclaimer: I dont believe this procedure if fully supported by VMware so please proceed with caution.

  • SSH to your primary vRA appliance
  • Run the following to view the contents of¬†/etc/vcac/server.xml
    • less /etc/vcac/server.xml
  • Look for the line with password= and copy everything between the “”. This password will allow you to connect to the vRA PostGres DB

  • Run the following command with the password from the above step
    • vcac-config prop-util -d –p “s2enc~K6RsAv5WGpoAt+qsnZPrKErxZ0kU1npeK/G5iMzyaWI=”
  • Next change to the postgres user
    • su postgres
  • Change to the postgres directory
    • cd /opt/vmware/vpostgres/current/bin
  • Connect to the vcac database
    • ./psql vcac -W
  • Enter the password from server.xml
  • vRA requests are store in the cat_request table. To enable us to delete a request we first need the request id. Query the cat_request table for your request ID using the requestnumber (In my case the offending failed requestnumber is 63, as seen in the first column in the screenshot above. replace with your requestnumber)
    • SELECT id,requestnumber FROM cat_request where requestnumber = ’63’;

vRA XaaS blueprint requests are referenced in 1 further table, cat_requestevent. This entriy must be deleted before you can delete the request.

  • Run the following commands to delete the request.
  • delete from cat_requestevent where request_id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;
  • delete from cat_request where id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;

The offending failed request should now be gone from the requests list in vRA!

Setup multiple vRA tenants using powershell and the vRA7 REST API

Following on from my post on Creating a local user in vRA7 using the REST API i wanted to try and script the entire process of creating multiple vRealize Automation 7 tenants as in our lab as we often need to spin up multiple tenants for testing or development purposes.

Some assumptions:

  • Each tenant has the same prefix of “dev-” and is appended with a 3 digit number starting at “001”
  • Each tenant gets the same local user created with matching credentials
  • Each tenant gets the same AD directory added
  • Each tenant gets the same AD groups added

This script will do the following:

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So as to avoid the requirement to edit the powershell script directly i put all configuration variables in an external .cfg file. This file needs to be placed in the same directory as the powershell script.

Firstly here is the config file contents. Edit each variable to match your environment. Modify the numberOfTenants variable to set the number of tenants you want to create. The example below will create 20 tenants.


[vRA Credentials to acquire authentication token]

[Create tenant details]

[Local Admin User Details]

[Tenant Directory Details]

[AD Domain Groups to add as Tenant & IaaS Admins]

And here is the script to create the tenants. It is broken up into multiple functions

# Script to create vRA7 Tenants in bulk
# Ensure you update the associated cfg file
# with the details of your vRA environment
# and details of the tenants you wish to create
# Created by Brian O'Connell
# Version 1.0.0

# Import configuration variables from external cfg file
Get-Content createvRATenants.cfg | Foreach-Object{
if ($_.length -gt 0) {
 $var = $_ -Split '=',2
 New-Variable -Name $var[0] -Value $var[1]

Function getvRAAuthToken {
# Construct credentials from config file
############# Get Auth token ###############
$Global:token = Invoke-RestMethod -Uri "https://$($VRA)/identity/api/tokens" -Method Post -Headers $headers -ContentType application/json -Body (ConvertTo-Json $credentials) | Select -ExpandProperty id
Write-Host "vRA Authentication Token Acquired" -ForegroundColor Green

Function createvRATenant {
 # ############ Create Tenant ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Create the Tenant
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
$tenantid = -join ($tenantIDPrefix,$tenantNumber)
$tenantURL = -join ($tenantURLPrefix,$tenantNumber)
$tenantName = -join ($tenantNamePrefix,$tenantNumber)
$tenantBody= @"
 "@type": "Tenant",
 "id": "$tenantid",
 "urlName": "$tenantURL",
 "name": "$tenantName",
 "description": "$tenantDescription",
 "contactEmail": "$tenantemailAddress"

$createTenant = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/tenants/$($tenantID)" -headers $headers -ContentType application/json -body $tenantBody
Write-Host "Tenant $($tenantName) created successfully" -ForegroundColor Green

Function createvRALocalAdminUser {
 ############# Create Local Admin User ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$userBody= @"
{ "@type": "User",
 "firstName": "$firstName",
 "lastName": "$lastName",
 "emailAddress": "$emailAddress",
 "description": "$description",
 "locked": false,
 "disabled": false,
 "password": "$password",
 "domain": "$domain",
 "userName": "$userName",
 "principalId": {
 "domain": "$domain",
 "name": "$name"

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the user
$createUser = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/principals" -headers $headers -ContentType "application/json" -body $userBody
Write-Host "Local Admin User for tenant $($tenantid) created successfully" -ForegroundColor Green


Function updatevRALocalAdminUserRoles {
 ############# Add Local Admin User to Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$principal = "vraadmin@vsphere.local"

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Add the user to tenant & IaaS admins
foreach ($roleID in $roleIDs) {
$makeUserAdmin = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($principal)/roles/$($roleID)" -headers $headers -body "{}"
Write-Host "Local Admin User Added to Tenant & IaaS Admins for tenant $($tenantid) " -ForegroundColor Green

Function createvRATenantDirectory {
 ############# Add AD Tenant directory ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

$directoryBody= @"
{"@type": "IdentityStore",
"domain": "$adDomain",
"name": "$adDomain",
"alias": "$adDomainalias",
"type": "$type",
"userNameDn": "$adUserNameDn",
"password": "$adBindPassword",
"url": "$adURL",
"groupBaseSearchDn": "$adGroupBaseSearchDn",
"userBaseSearchDn": "$adUserBaseSearchDn"
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the directory
$createDirectory = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/directories" -headers $headers -ContentType "application/json" -body $directoryBody
Write-Host "Tenant Directory Created for tenant $($tenantid) " -ForegroundColor Green


Function addDomainGroupstovRAAdmins {
############## Add AD Domain Groups to vRA Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Add the user to tenant & IaaS admins
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
$addTenantAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($tenantAdmins)/roles/$($tenantRoleID)" -headers $headers -body "{}"

$addIaaSAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($iaasAdmins)/roles/$($iaasRoleID)" -headers $headers -body "{}"
Write-Host "Domain groups added to as tenant & IaaS admins for tenant $($tenantid) " -ForegroundColor Green

# Call All functions to setup tenants
getvRAAuthToken; createvRATenant; createvRALocalAdminUser; updatevRALocalAdminUserRoles; createvRATenantDirectory; addDomainGroupstovRAAdmins 

Distributed vRA validation script

From time to time a distributed vRA deployment can have issues…here is a quick script to verify and validate the important components are up and functioning…without the need to log into multiple components. Here is a diagram of my distributed vRA setup

Distributed vRA v4

And here is the script! It has multiple functions to do the following

  • Basic ping tests to each component
  • Get the status of all vRA component services
  • Test the Web & manager server URLs

# Script to check the status of each component of a distributed vRA deployment
# Modify the hostnames to match your environment
$vRAAppliance01FQDN = "vra01.domain.local"
$vRAAppliance02FQDN = "vra02.domain.local"
$vRAWeb01FQDN = "web01.domain.local"
$vRAWeb02FQDN = "web02.domain.local"
$vRAManager01FQDN = "manager01.domain.local"
$vRAManager02FQDN = "manager02.domain.local"
$vRADEM01FQDN = "demw01.domain.local"
$vRADEM02FQDN = "demw02.domain.local"
$vRAAgent01FQDN = "agent01.domain.local"
$vRAAgent02FQDN = "agent02.domain.local"
$vRAComponentServiceURL = "https://vra-vip.domain.local/component-registry/services/status/current"
$webVIPURL = "https://web-vip.domain.local/WAPI"
$managerVIPURL = "https://manager-vip.domain.local/VMPS2"


Function pingHosts {
Performing basic ping test to each defined component
$vms = @($vRAAppliance01FQDN, $vRAAppliance02FQDN, $vRAWeb01FQDN, $vRAWeb02FQDN, $vRAManager01FQDN, $vRAManager02FQDN, $vRADEM01FQDN, $vRADEM02FQDN, $vRAAgent01FQDN, $vRAAgent02FQDN)
$collection = $()
foreach ($vm in $vms)
 $status = @{ "ServerName" = $vm; "TimeStamp" = (Get-Date -f s) }
 if (Test-Connection $vm -Count 1 -ea 0 -Quiet)
 $status["Ping Results"] = "Up"
 $status["Ping Results"] = "Down" 
 New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
 $collection += $serverStatus

Function testvRAServices {
Getting Status of all vRA component services
Write-Host "Checking status of vRA Component Services" -ForegroundColor Yellow
# Request Service Information from $vRAComponentServiceURL
$vRAURL = Invoke-WebRequest $vRAComponentServiceURL
# Convert the Json response
$json = ConvertFrom-Json -InputObject $vRAURL.content
# Get Service name & status
$serviceInfo = $json.content
# Loop through each service
foreach ($service in $serviceInfo) {
# Get the Service Name
$serviceName = $service.serviceName
# Get the Service status
$serviceStatus = $service.serviceStatus.serviceInitializationStatus
# If Service Status is blank report it as BLANK POSSIBLY STOPPED
 if (!$serviceStatus) {
 $serviceStatus = "BLANK - POSSIBLY STOPPED?"
# If Service Status is FAILED print to screen in red 
if ($serviceStatus -eq "FAILED") {
 write-host "$serviceName $serviceStatus" -ForeGroundColor Red
# Otherwise print to screen as normal (Remove this if you only want to report failed services) 
 else {
 Write-Host "$serviceName $serviceStatus"}


Function testWebVIP {
Checking status of vRA Web API URL
Write-Host "Testing IaaS Web Service VIP URL $webVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($webVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Web Service is OK!" -ForegroundColor Green
 # Close HTTP request
Else {
 Write-Host "IaaS Web Service is not responding. Restart IIS on Web01. If that does not resolve then Reboot Web01" -ForegroundColor Red
Function testMgrVIP {
Checking status of vRA Manager API URL
Write-Host "Testing IaaS Manager Service VIP URL $managerVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($managerVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Manager Service is OK!" -ForegroundColor Green
 # Close HTTP request
Else {
 Write-Host "IaaS Manager Service is not responding. Ensure all vRA services are running on manager01. If that does not resolve then Reboot manager01" -ForegroundColor Red

 pingHosts; testvRAServices; testWebVIP; testMgrVIP

The function pingHosts is a basic ping test to each defined vRA component

The function testvRAServices was an interesting one to write as I’m not overly familiar with working with APIs so it was a learning experience. I wanted to be able to report the status of all vRA services listed on the VAMI administration UI of a vRA appliance (https://vra:5480). The URL that the services are listed on is https://vra-vip.domain.local/component-registry/services/status/current so using the powershell Invoke-WebRequest you get back the page information.


Line 56 in the script puts the page contents into a variable we can work with. You can see that the information we want is stored in Content (ServiceStatus) in Json format so you need to take that Json and convert it to  to powershell readable text using the ConvertFrom-Json function (ConvertFrom-Json converts a JSON-formatted string to a custom object (PSCustomObject) that has a property for each field in the JSON string) Line 58 does this

We then put each service into the $serviceinfo variable and loop through them to get the service name and service status.

Side note here: Originally I was querying $json.content.serviceStatus to get the details i wanted but i noticed I wasnt getting a full list of services, i was getting some blank content and also some duplicate service names. This is how i was doing it

$vRAURL = Invoke-WebRequest "https://vra-vip.vlab.local/component-registry/services/status/current" 
$json = ConvertFrom-Json -InputObject $vRAURL.content 
$serviceInfo = $json.content.serviceStatus | Select serviceName,serviceInitializationStatus $serviceInfo 

Here is that that returns. As you can see its not the full list and there is a duplicate entry so its not much use

Duplicate Results

I dug a little into the API and it seems that it does indeed contain inconsistent information. Here is an excerpt with some issues where the actual service name is content-management but the serviceStatus reports the name as advanced-designer-service

Service Name issue

So to get an accurate list i queried the serviceName field to get the name and the serviceStatus.serviceInitializationStatus to get the service status. Unfortunately doing it this way doesnt allow creating a nice formatted table (at least i havent figured out how to do it yet!) but i did get it to print out each service & status on the same line.

Line 68: In my lab i use a vRO appliance so the internal vRO service on the vRA appliance is stopped. The service status comes back blank so i added a check to report blank service status as “BLANK – POSSIBLY STOPPED?”.

Line 72: I also added a check to print any failed services in red so they stand out.

The testWebVIP and testManagerVIP functions use the powershell System.Net.WebRequest to get the HTTP status code for a given URL. If the status code comes back as 200 then everything is ok. If not there is an issue with your IaaS components

So there you have it. A quick way to verify the status of all of the important vRealize Automation components and services. In my example below the iaas-service is in a failed state (The driving reason for creating this script! ūüôā )

Script Results

Failed vRA IaaS Web Server Install

There are many reasons why a vRealize Automation IaaS Web server install can fail

  • MSDTC Issues
  • NTP Issues
  • DNS Issues
  • Certificate Issues
  • Mental Issues…. (Caused by all of the above!)

I hit this error in the lab and the fix was a new one to me so said i’d document it. The IaaS web server was failing to install and the error in the logs is below which pointed to the certs

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

I checked and double checked all certificates and SAN attributes of each component, checked DNS resolution forward and reverse, verified NTP etc. Then it occurred to me that the signing CA is from a different domain than the domain i was installing in so i added the root cert to the local trusted root certification store and low and behold the install completed successfully. Had to do the same on all other IaaS VMs. So even though the root and signing certs were in the personal store with the SAN certs it was not sufficient.


vRA 6.2 Distributed IaaS deployment

I needed to deploy a distributed vRA 6.2 IaaS in the lab and didn’t find too many resources on the web so here are some of my notes! I hope to revisit this soon to do a full distributed HA deployment including NSX load balancers. For now here is a standard distributed deployment


  • AD installed & available
  • vCenter/SSO installed & available
  • SQL Server installed and available and IaaS SQL pre reqs run for MSDTC
  • vRA 6.2 appliance deployed and integrated with SSO
  • AD CA configured (Will try to cover this in another post)
  • 5 Windows 2012 R2 VMs. 1 for each of the following roles – Web Server, Manager Server, DEM Orchestrator, DEM Worker, IaaS Agent
  • All windows VMs joined the AD domain
  • Run Brian Graf’s¬†IaaS pre req script¬†on all VMs (Pretty sure its only required on the Web & manager server but didnt take any chances!)

Here are the steps

  • Download the IaaS installer from the vRA appliance @ https://vra-FQDN:5480/i
  • First we need to install the database component. From any VM run the IaaS installer
  • Accept the ELUA and click Next
  • Enter the root credentials for the vRA appliance and select Accept Certificate and click Next
  • Choose Custom Install > IaaS Server

Installation Type

  • Select Database from the¬†features list and enter the database instance and database name and click Next (I’m using SQL Express in the lab)

2.install DB

  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

3.DB Pre Reqs

  • Click Install on the Ready to Install Screen and Next and Finish once the Database install completes
  • Next you need to install the Website feature
  • Before installing the website component it is recommended to generate a CA signed cert for the server FQDN.
  • To do this open IIS on the Web server, select your server name and select Server Certificates
  • In the right hand pane select Create Domain Certificate

Generate Certificate

  • Enter the certificate distinguished name for the web server and click Next

Certificate Request


  • Click Select to choose your CA and assign a friendly name (user the web server FQDN)
  • Once complete¬†run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Website from the feature list.
  • Choose the Default Web Site and the default port of 443
  • Under Available certificates choose the CA signed certificate you created earlier and enter the web server FQDN for the IaaS Server

web server setup

  • Because the IaaS Model Manager Data is required for the Website component to run you must also select the ModelManagerData feature
  • Enter the vCAC/vRA appliance FQDN & SSO details and also the IaaS Web Server FQDN and click Next

9. ModelManagerData


  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

10. ModelManagerPreReqs

  • On the Server and Account Settings screen enter the password for the service account being used and a passphrase for the database and click Next and click Finish once the install completes

11. ModelManagerData Account settings

  • Next up is the Manager Service
  • On the VM you designated for the¬†Manager Service follow the steps outlined earlier to generate a CA signed certificate
  • Once complete¬†run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Manager Service¬†from the feature list.
  • Enter the FQDN of the Web Server and select the certificate you created in the previous steps and click Next and click Finish

managerService Install


  • Next up is the DEM Orchestrator
  • On the VM you designated for the DEMO Orchestrator run the IaaS installer, choose custom > Distributed Execution Managers and click Next
  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Distributed Execution Manager Screen do the following
    • From the DEM role drop down choose Orchestrator
    • Enter a name
    • Enter a description
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Click Add and click Next and click Install and then click Finish

DEM Orchestratot Config


  • Repeat the above steps on the DEM Worker VM choosing the DEM Worker Role
  • Finally we will install a Proxy Agent that will be used to communicate with vRA endpoints
  • On the VM you designated for the IaaS Agent¬†run the IaaS installer, choose custom >¬†Proxy Agents and click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Proxy Agent¬†Screen do the following
    • From the Agent Type¬†drop down choose¬†vSphere
    • Enter a name
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Enter the vCenter Endpoint FQDN
    • Click Add and click Next and click Install and then click Finish

Proxy Agent Install

  • If everything went according to plan you should now be able to log into vRA and configure tenants and resources and start using your distributed IaaS installation.

Hopefully if you’ve read this far you found this useful. I hope to post the distributed HA procedure soon once i get some free lab time!