Cleanup Failed Tasks in SDDC Manager

I was chatting with my colleague Paudie O’Riordan yesterday about PowerVCF as he was doing some testing internally and he mentioned that a great addition would be to have the ability to find, and cleanup failed tasks in SDDC Manager. Some use cases for this would be, cleaning up an environment before handing it off to a customer, or before recording a demo etc.

Currently there isnt a supported public API to delete a failed task so you have to run a curl command on SDDC Manager with the task ID. So getting a list of failed tasks and then running a command to delete each one can take time. See Martin Gustafson’s post on how to do it manually here.

I took a look at our existing code for retrieving tasks (and discovered a bug in the logic that is now fixed in PowerVCF 2.1.5!) and we have the ability to specify -status. So requesting a list of tasks with -status “failed” returns a list. So i put the script below together to retrieve a list of failed tasks, loop through them and delete them. The script requires the following inputs

  • SDDC Manager FQDN. This is the target that is queried for failed tasks
  • SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
  • Password for the above user
  • Password for the SDDC Manager appliance vcf user. This is used to run the task deletion. This is not tracked in the credentials DB so we need to pass it.

Once the above variables are populated the script does the following:

  • Checks for PowerVCF (minimum version 2.1.5) and installs if not present
  • Requests an API token from SDDC Manager
  • Queries SDDC Manager for the management domain vCenter Server details
  • Uses the management domain vCenter Server details to retrieve the SDDC Manager VM name
  • Queries SDDC Manager for a list of tasks in a failed state
  • Loops through the list of failed tasks and deletes them from SDDC Manager
  • Verifies the task is no longer present

Here is the script. It is also published here if you would like to enhance it

# Script to cleanup failed tasks in SDDC Manager
# Written by Brian O'Connell - Staff Solutions Architect @ VMware

#User Variables
# SDDC Manager FQDN. This is the target that is queried for failed tasks
$sddcManagerFQDN = ""
# SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
$sddcManagerAPIUser = "administrator@vsphere.local"
$sddcManagerAPIPassword = "VMw@re1!"
# Password for the SDDC Manager appliance vcf user. This is used to run the task deletion
$sddcManagerVCFPassword = "VMw@re1!"


# Set TLS to 1.2 to avoid certificate mismatch errors
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# Install PowerVCF if not already installed
if (!(Get-InstalledModule -name PowerVCF -MinimumVersion 2.1.5 -ErrorAction SilentlyContinue)) {
    Install-Module -Name PowerVCF -MinimumVersion 2.1.5 -Force

# Request a VCF Token using PowerVCF
Request-VCFToken -fqdn $sddcManagerFQDN -username $sddcManagerAPIUser -password $sddcManagerAPIPassword

# Disconnect all connected vCenters to ensure only the desired vCenter is available
if ($defaultviservers) {
    $server = $defaultviservers.Name
    foreach ($server in $defaultviservers) {            
        Disconnect-VIServer -Server $server -Confirm:$False

# Retrieve the Management Domain vCenter Server FQDN
$vcenterFQDN = ((Get-VCFWorkloadDomain | where-object {$_.type -eq "MANAGEMENT"}).vcenters.fqdn)
$vcenterUser = (Get-VCFCredential -resourceType "PSC").username
$vcenterPassword = (Get-VCFCredential -resourceType "PSC").password

# Retrieve SDDC Manager VM Name
if ($vcenterFQDN) {
    Write-Output "Getting SDDC Manager Manager VM Name"
    Connect-VIServer -server $vcenterFQDN -user $vcenterUser -password $vcenterPassword | Out-Null
    $sddcmVMName = ((Get-VM * | Where-Object {$_.Guest.Hostname -eq $sddcManagerFQDN}).Name)              

# Retrieve a list of failed tasks
$failedTaskIDs = @()
$ids = (Get-VCFTask -status "Failed").id
Foreach ($id in $ids) {
    $failedTaskIDs += ,$id
# Cleanup the failed tasks
Foreach ($taskID in $failedTaskIDs) {
    $scriptCommand = "curl -X DELETE$taskID"
    Write-Output "Deleting Failed Task ID $taskID"
    $output = Invoke-VMScript -ScriptText $scriptCommand -vm $sddcmVMName -GuestUser "vcf" -GuestPassword $sddcManagerVCFPassword

# Verify the task was deleted    
    Try {
    $verifyTaskDeleted = (Get-VCFTask -id $taskID)
    if ($verifyTaskDeleted -eq "Task ID Not Found") {
        Write-Output "Task ID $taskID Deleted Successfully"
    catch {
        Write-Error "Something went wrong. Please check your SDDC Manager state"
Disconnect-VIServer -server $vcenterFQDN -Confirm:$False

As always, comments/feedback welcome!

Part 2: Working With the SRM VAMI API : Replacing the Appliance Certificate

In Part 1 of this series we saw how to retrieve a sessionId from the Site Recovery Manager VAMI interface using Postman & Powershell. In this post we will use that sessionId to replace the appliance SSL certificate using the API. To start we again use the VAMI UI to inspect the endpoint URL being used for certificate replacement by doing a manual replacement. In this case the URL is:

Site Recovery Manager expects the certificate in P12 format so I used CertGen to create the cert format needed. When using the UI you browse to the cert file and it uploads in the browser along with the certificate passphrase. Behind the scenes it is then base64 encoded, so you need to do this before using the API.

# Base64 encoded the p12 file

$certFile = ".\sfo-m01-srm01.4.p12"

$base64string = [Convert]::ToBase64String([IO.File]::ReadAllBytes($certFile))

$body = '{
"certificateContent": "'+$base64string+'",
"certificatePassword": "'+$certPassword+'"

#Create the required headers using the sessionId

$headers = @{"Content-Type" = "application/json"}
$headers.Add("dr.config.service.sessionid", "$sessionId")

$uri = ""

Invoke-RestMethod -Method POST -Uri $uri -Headers $headers -body $body

And there you have it..your appliance cert replaced via the API.

Part 1: Working With the SRM VAMI API : Retrieving a Session ID

I’ve recently been doing a lot of work with VMware Site Recovery Manager (SRM) and vSphere Replication (vSR) with VMware Cloud Foundation. Earlier this year we (Ken Gould & I) published an early access design for Site Protection & Recovery for VMware Cloud Foundation 4.2. We have been working to refresh & enhance this design for a new release. Part of this effort includes trying to add some automation to assist with the manual steps to speed up time to deploy. SRM & vSR do not have publicly documented VAMI APIs so we set about trying to automate the configuration with a little bit of reverse engineering.

As with most APIs, whether public or private, you must authenticate before you can run an API workflow, so the first task is figuring out how the authentication to perform a workflow works. Typically, if you hit F12 in your browser you will get a developer console that exposes what goes on behind the scenes in a browser session. So to inspect the process, use the browser to perform a manual login, and review the header & response tabs in the developer view. This exposes the Request URL to use, the method (POST) and the required headers (accept: application/json)

The Response tab shows a sessionId which can be used for further configuration API calls in the headers as dr.config.service.sessionid

So with the above information you can use an API client like Postman to retrieve a sessionId with the URL & headers like this

And your VAMI admin user and password in JSON format in the body payload

You can also use the information to retrieve a sessionId using PowerShell

$headers = @{"Content-Type" = "application/json"}
$uri = ""
$body = '{"username": "admin","password": "mypassword"}'

$request = Invoke-RestMethod -Method POST -Uri $uri -Headers $headers -body $body

$sessionID = $


Keep an eye out for additional posts where we will use the sessionId to perform API based tasks

Checking Password Expiry For VMware Cloud Foundation Management Components

Within a VMware Cloud Foundation instance, SDDC Manager is used to manage the lifecycle of passwords (or credentials). While we provide the ability to rotate (either scheduled or manually) currently there is no easy way to check when a particular password is due to expire, which can lead to appliance root passwords expiring, which will cause all sorts of issues. The ability to monitor expiry is something that is being worked on, but as a stop gap I put together the script below which leverages PowerVCF and also a currently undocumented API for validating credentials.

The script has a function called Get-VCFPasswordExpiry that accepts the following parameters

  • -fqdn (FQDN of the SDDC Manager)
  • -username (SDDC Manager Username – Must have the ADMIN role)
  • -password (SDDC Manager password)
  • -resourceType (Optional parameter to specify a resourceType. If not passed, all resources will be checked. If passed (e.g. VCENTER) then only that resourceType will be checked. Supported resource types are

PowerVCF is a requirement. If you dont already have it run the following

Install-Module -Name PowerVCF

The code takes a while to run as it needs to do the following to check password expiry

  • Connect to SDDC Manager to retrieve an API token
  • Retrieve a list of all credentials
  • Using the resourceID of each credential
    • Perform a credential validation
    • Wait for the validation to complete
    • Parse the results for the expiry details
    • Add all the results to an array and present in a table (Kudos to Ken Gould for assistance with the presentation of this piece!)

In this example script I am returning all non SERVICE user accounts regardless of expiry (SERVICE account passwords are system managed). You could get more granular by adding something like this to only display accounts with passwords due to expire in less than 14 days

if ($validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry -lt 14) {
               Write-Output "Password for username $($validationTaskResponse.validationChecks.username) expires in $($validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry) days"

Here is the script content. As always feedback is welcome. Also posted in Github here if anyone wants to fork and improve

# Script to check the password expiry of VMware Cloud Foundation Credentials
# Written by Brian O'Connell - VMware

#User Variables
$sddcManagerFQDN = ""
$sddcManagerAdminUser = "administrator@vsphere.local"
$sddcManagerAdminPassword = "VMw@re1!"

# Requires PowerVCF Module
#Requires -Module PowerVCF

Function Get-VCFPasswordExpiry

    Param (
        [Parameter (Mandatory = $true)] [ValidateNotNullOrEmpty()] [String]$fqdn,
        [Parameter (Mandatory = $true)] [ValidateNotNullOrEmpty()] [String]$username,
        [Parameter (Mandatory = $true)] [ValidateNotNullOrEmpty()] [String]$password,
        [Parameter (Mandatory = $false)] [ValidateSet("VCENTER", "PSC", "ESXI", "BACKUP", "NSXT_MANAGER", "NSXT_EDGE", "VRSLCM", "WSA", "VROPS", "VRLI", "VRA", "VXRAIL_MANAGER")] [ValidateNotNullOrEmpty()] [String]$resourceType
# Request an SDDC manager Token
Request-VCFToken -fqdn $fqdn -username $username -password $password
# Build the required headers
$credentialheaders = @{"Content-Type" = "application/json"}
$credentialheaders.Add("Authorization", "Bearer $accessToken")
# Get all credential objects that are not type SERVICE
if (!$PsBoundParameters.ContainsKey("resourceType")) {
$credentials = Get-VCFCredential | where-object {$_.accountType -ne "SERVICE"}
else {
    $credentials = Get-VCFCredential -resourceType $resourceType | where-object {$_.accountType -ne "SERVICE"}
$validationArray = @()
Foreach ($credential in $credentials) {
    $resourceType = $credential.resource.resourceType
    $resourceID = $credential.resource.resourceId
    $username = $credential.username
    $credentialType = $credential.credentialType
    $body = '[
        "resourceType": "'+$resourceType+'",
        "resourceId": "'+$resourceID+'",
        "credentials": [
                "username": "'+$username+'",
                "credentialType": "'+$credentialType+'"
    $uri = "https://$sddcManagerFQDN/v1/credentials/validations"
    # Submit a credential validation request
            $response = Invoke-RestMethod -Method POST -URI $uri -headers $credentialheaders -body $body
            $validationTaskId = $

            Do {
                # Keep checking until executionStatus is not IN_PROGRESS
                $validationTaskuri = "https://$sddcManagerFQDN/v1/credentials/validations/$validationTaskId"
                $validationTaskResponse = Invoke-RestMethod -Method GET -URI $validationTaskuri -headers $credentialheaders
            While ($validationTaskResponse.executionStatus -eq "IN_PROGRESS")
            # Build the output
            $validationObject = New-Object -TypeName psobject
            $validationObject | Add-Member -notepropertyname 'Resource Name' -notepropertyvalue $validationTaskResponse.validationChecks.resourceName
            $validationObject | Add-Member -notepropertyname 'Username' -notepropertyvalue $validationTaskResponse.validationChecks.username
            $validationObject | Add-Member -notepropertyname 'Number Of Days To Expiry' -notepropertyvalue $validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry
            Write-Output "Checking Password Expiry for username $($validationTaskResponse.validationChecks.username) from resource $($validationTaskResponse.validationChecks.resourceName)"
            # Add each credential result to the array
            $validationArray += $validationObject
# Print the array

# Run the function
Get-VCFPasswordExpiry -fqdn $sddcManagerFQDN -username $sddcManagerAdminUser -password $sddcManagerAdminPassword

# Run the function with resourceType VCENTER
# Get-VCFPasswordExpiry -fqdn $sddcManagerFQDN -username $sddcManagerAdminUser -password $sddcManagerAdminPassword -resourceType VCENTER

Here is a screenshot of the result

Announcing PowerVCF 2.0

I’m happy to announce the availability of PowerVCF 2.0. This version of PowerVCF is compatible with VMware Cloud Foundation 4.0 and above. Due to some API security enhancements in VCF 4.0 around the use of API tokens for authentication the module has been refactored to leverage access & refresh tokens (more on that here). For that reason if you would like to use PowerVCF for VCF 3.9.x you should continue to use PowerVCF 1.2 .

PowerVCF 2.0 is published to the PowerShell Gallery here

Whats new in PowerVCF 2.0

Along with a number of new or modified cmdlets the following enhancements have been made:

  • Grouped cmdlets based on order of API documentation
  • Code hygiene

The following table provides a detailed breakdown of all the changes for this release. Thanks to my colleague @GaryJBlake for doing most of the work on this release and for pulling this list together!

Category cmdlet Name Description Comment
Backup and Restore Start-VCFRestore Starts the restore process of SDDC Manager NEW
Backup and Restore Get-VCFRestoreTasks Gets the status of the restore process NEW
Connectivity Connect-VCFManager Create authentication header for SDDC Manager appliance UPDATED – Support the new token / bearer authentication model and basicAuth switch for restore process
Connectivity Connect-CloudBuilder Create authentication header for Cloud Builder appliance NEW
Certificates Get-VCFCertificateAuthority Get Certificate Authority information UPDATED – Added support for getting the details by id
Certificates Remove-VCFCertificateAuthority Deletes Certificate Authority configuration NEW
Certificates Get-VCFCertificate View certificate of all the resources in a domain UPDATED – Added support for get certificate details by resource
Credentials Get-VCFCredential Get the credentials UPDATED- Added support for getting the details by id
Credentials Stop-VCFCredentialTask Cancels a failed update or rotate passwords task RENAMED – From Cancel-VCFCredentialTask
Credentials Restart-VCFCredentialTask Retry a failed rotate/update passwords task RENAMED – From Retry-VCFCredentialTask
Hosts Commission-VCFHost Commissions a list of hosts UPDATED – Added support for validating the input spec for host operations (-validate switch)
NSX-T Edge Clusters Get-VCFEdgeCluster Get an Edge Cluster NEW
NSX-T Edge Clusters New-VCFEdgeCluster creates an NSX-T edge cluster NEW
Personalities Get-VCFPersonality Get the vSphere Lifecycle Manager Personalities NEW
SDDC (Cloud Builder) Get-CloudBuilderSDDC Retrieve all SDDCs NEW
SDDC (Cloud Builder) Start-CloudBuilderSDDC Create SDDC NEW
SDDC (Cloud Builder) Restart-CloudBuilderSDDC Retry failed SDDC creation NEW
SDDC (Cloud Builder) Get-CloudBuilderSDDCValidation Get all SDDC specification validations NEW
SDDC (Cloud Builder) Start-CloudBuilderSDDCValidation Validate SDDC specification before creation NEW
SDDC (Cloud Builder) Stop-CloudBuilderSDDCValidation Cancel SDDC specification validation NEW
SDDC (Cloud Builder) Restart-CloudBuilderSDDCValidation Retry SDDC validation NEW
System Prechecks Start-VCFSystemPrecheck Perform System Precheck RENAMED – From Start-PreCheckVCFSystem
System Prechecks Get-VCFSystemPrecheckTask Get System Precheck Task RENAMED – From Get-PreCheckVCFSystemTask
Tasks Restart-VCFTask Retry a previously failed task RENAMED – From Retry-VCFTask
Users Get-VCFRole Get all roles NEW
Users Get-VCFUser Get all Users NEW
Users New-VCFUser Adds a new user NEW
Users New-VCFServiceUser Adds a new service user NEW
Users Delete-User Deletes a user NEW
vRealize Suite Lifecycle Manager Reset-VCFvRSLCM Redeploy vRealize Suite Lifecycle Manager NEW
vRealize Suite Lifecycle Manager New-VCFvRSLCM Validate the input specification for vRealize Suite Lifecycle Manager deployment UPDATED – Added support for validating the json spec (-validate switch).

Automate your VMware Validated Design NSX-V Distributed Firewall Configuration

A few weeks back I mentioned on twitter that i was working on automating the VMware Validated Design NSX-V Distributed Firewall Configuration in my lab. (I admit it took longer than i had planned!) Currently this is a manual post deployment step once VMware Cloud Builder has completed the deployment. This will likely be picked up by Cloud Builder in a future release but for now its a manual, and somewhat tedious, but required, step!

Full details on the manual steps required for this configuration can be found here. Please take the time to understand what these rules are doing before implementing them.

So in an effort to make this post configuration step a little less painful i set out to automate it. I’ve played with the NSX-V API in the past and found it much easier to interact with by using PowerNSX, rather than leveraging PostMan and the API directly. PowerNSX is the unofficial, official automation tool for NSX. Hats off to VMware engineers Nick Bradford, Dale Coghlan & Anthony Burke for creating and documenting this tool. Anthony also published a FREE book on Automating NSX for vSphere with PowerNSX. More on that here.

Disclaimer: This script is not officially supported by VMware. Use at your own risk & test in a development/lab environment before using in production.

I’ve posted the script to GitHub here as its a bit lengthy! There may be a more efficient way to do some parts of it and if anyone wants to contribute please feel free!

As with a lot of the scripts i create it is menu based and has 2 main options:

  1. Create DFW exclusions, IP Sets & Security Groups
  2. Create DFW Rules

The reason i split it into 2 distinct operations is to allow you to inspect the exclusion list, IP Sets & Security Groups before creating the firewall rules. This will ensure that you dont lock yourself out of vCenter by creating an incorrect rule.

Required Software

  • PowerCli
    • The script will check for PowerCli and if not found will attempt to install the latest version from the PowerShell Gallery
    • Currently tested on Windows only
    • If you dont have internet access you can manually install PowerCli by opening a PowerShell console as administrator and running:
    • Find-Module -Name VMware.PowerCLI | Install-Module
  • PowerNSX
    • The script will check for PowerNSX and if not found will attempt to install the latest version from the PowerShell Gallery
    • Currently tested on Windows only
    • If you dont have internet access you can manually install PowerNSX by opening a PowerShell console as administrator and running:
    • Find-Module -Name PowerNSX | Install-Module

Required Variables

Before you can run the script you need to edit the User Variables to provide the following:

  • Target vCenter details
    • Required to establish a PowerCli Connection with vCenter Server. This is used when updating the DFW exclusion list
  • Target NSX Manager details
    • Required to establish a connection with NSX manager to configure the DFW
  • IP Addresses for the various SDDC components

Hopefully you will find this useful!

Distributed vRA validation script

From time to time a distributed vRA deployment can have issues…here is a quick script to verify and validate the important components are up and functioning…without the need to log into multiple components. Here is a diagram of my distributed vRA setup

Distributed vRA v4

And here is the script! It has multiple functions to do the following

  • Basic ping tests to each component
  • Get the status of all vRA component services
  • Test the Web & manager server URLs

# Script to check the status of each component of a distributed vRA deployment
# Modify the hostnames to match your environment
$vRAAppliance01FQDN = "vra01.domain.local"
$vRAAppliance02FQDN = "vra02.domain.local"
$vRAWeb01FQDN = "web01.domain.local"
$vRAWeb02FQDN = "web02.domain.local"
$vRAManager01FQDN = "manager01.domain.local"
$vRAManager02FQDN = "manager02.domain.local"
$vRADEM01FQDN = "demw01.domain.local"
$vRADEM02FQDN = "demw02.domain.local"
$vRAAgent01FQDN = "agent01.domain.local"
$vRAAgent02FQDN = "agent02.domain.local"
$vRAComponentServiceURL = "https://vra-vip.domain.local/component-registry/services/status/current"
$webVIPURL = "https://web-vip.domain.local/WAPI"
$managerVIPURL = "https://manager-vip.domain.local/VMPS2"


Function pingHosts {
Performing basic ping test to each defined component
$vms = @($vRAAppliance01FQDN, $vRAAppliance02FQDN, $vRAWeb01FQDN, $vRAWeb02FQDN, $vRAManager01FQDN, $vRAManager02FQDN, $vRADEM01FQDN, $vRADEM02FQDN, $vRAAgent01FQDN, $vRAAgent02FQDN)
$collection = $()
foreach ($vm in $vms)
 $status = @{ "ServerName" = $vm; "TimeStamp" = (Get-Date -f s) }
 if (Test-Connection $vm -Count 1 -ea 0 -Quiet)
 $status["Ping Results"] = "Up"
 $status["Ping Results"] = "Down" 
 New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
 $collection += $serverStatus

Function testvRAServices {
Getting Status of all vRA component services
Write-Host "Checking status of vRA Component Services" -ForegroundColor Yellow
# Request Service Information from $vRAComponentServiceURL
$vRAURL = Invoke-WebRequest $vRAComponentServiceURL
# Convert the Json response
$json = ConvertFrom-Json -InputObject $vRAURL.content
# Get Service name & status
$serviceInfo = $json.content
# Loop through each service
foreach ($service in $serviceInfo) {
# Get the Service Name
$serviceName = $service.serviceName
# Get the Service status
$serviceStatus = $service.serviceStatus.serviceInitializationStatus
# If Service Status is blank report it as BLANK POSSIBLY STOPPED
 if (!$serviceStatus) {
 $serviceStatus = "BLANK - POSSIBLY STOPPED?"
# If Service Status is FAILED print to screen in red 
if ($serviceStatus -eq "FAILED") {
 write-host "$serviceName $serviceStatus" -ForeGroundColor Red
# Otherwise print to screen as normal (Remove this if you only want to report failed services) 
 else {
 Write-Host "$serviceName $serviceStatus"}


Function testWebVIP {
Checking status of vRA Web API URL
Write-Host "Testing IaaS Web Service VIP URL $webVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($webVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Web Service is OK!" -ForegroundColor Green
 # Close HTTP request
Else {
 Write-Host "IaaS Web Service is not responding. Restart IIS on Web01. If that does not resolve then Reboot Web01" -ForegroundColor Red
Function testMgrVIP {
Checking status of vRA Manager API URL
Write-Host "Testing IaaS Manager Service VIP URL $managerVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($managerVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Manager Service is OK!" -ForegroundColor Green
 # Close HTTP request
Else {
 Write-Host "IaaS Manager Service is not responding. Ensure all vRA services are running on manager01. If that does not resolve then Reboot manager01" -ForegroundColor Red

 pingHosts; testvRAServices; testWebVIP; testMgrVIP

The function pingHosts is a basic ping test to each defined vRA component

The function testvRAServices was an interesting one to write as I’m not overly familiar with working with APIs so it was a learning experience. I wanted to be able to report the status of all vRA services listed on the VAMI administration UI of a vRA appliance (https://vra:5480). The URL that the services are listed on is https://vra-vip.domain.local/component-registry/services/status/current so using the powershell Invoke-WebRequest you get back the page information.


Line 56 in the script puts the page contents into a variable we can work with. You can see that the information we want is stored in Content (ServiceStatus) in Json format so you need to take that Json and convert it to  to powershell readable text using the ConvertFrom-Json function (ConvertFrom-Json converts a JSON-formatted string to a custom object (PSCustomObject) that has a property for each field in the JSON string) Line 58 does this

We then put each service into the $serviceinfo variable and loop through them to get the service name and service status.

Side note here: Originally I was querying $json.content.serviceStatus to get the details i wanted but i noticed I wasnt getting a full list of services, i was getting some blank content and also some duplicate service names. This is how i was doing it

$vRAURL = Invoke-WebRequest "https://vra-vip.vlab.local/component-registry/services/status/current" 
$json = ConvertFrom-Json -InputObject $vRAURL.content 
$serviceInfo = $json.content.serviceStatus | Select serviceName,serviceInitializationStatus $serviceInfo 

Here is that that returns. As you can see its not the full list and there is a duplicate entry so its not much use

Duplicate Results

I dug a little into the API and it seems that it does indeed contain inconsistent information. Here is an excerpt with some issues where the actual service name is content-management but the serviceStatus reports the name as advanced-designer-service

Service Name issue

So to get an accurate list i queried the serviceName field to get the name and the serviceStatus.serviceInitializationStatus to get the service status. Unfortunately doing it this way doesnt allow creating a nice formatted table (at least i havent figured out how to do it yet!) but i did get it to print out each service & status on the same line.

Line 68: In my lab i use a vRO appliance so the internal vRO service on the vRA appliance is stopped. The service status comes back blank so i added a check to report blank service status as “BLANK – POSSIBLY STOPPED?”.

Line 72: I also added a check to print any failed services in red so they stand out.

The testWebVIP and testManagerVIP functions use the powershell System.Net.WebRequest to get the HTTP status code for a given URL. If the status code comes back as 200 then everything is ok. If not there is an issue with your IaaS components

So there you have it. A quick way to verify the status of all of the important vRealize Automation components and services. In my example below the iaas-service is in a failed state (The driving reason for creating this script! 🙂 )

Script Results

Create a local Tenant user in vRA7 using the rest API

VMware vRealize Automation 7.x is the latest version of VMware’s cloud automation software. With the new version comes a change in the way tenants (and user roles for same) are setup. In previous versions you would do the following to create a tenant

  • Log into the default tenant
  • Create a new tenant
  • Add an identity store
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

With vRA 7.x the procedure changes (and becomes more cumbersome IMO)

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So when it comes to trying to automate tenant creation its not as easy as it used to be for 6.x. For 6.x I’ve used cloud client (if you’re not familiar with CloudClient i’d recommend checking it out here )

The CloudClient commands required for 6.x are as follows:

  • Firstly login to CloudClient
vra login userpass --server vra-vip.domain.local --user administrator@vsphere.local --password Password123! --tenant vsphere.local
  • Next create the tenant
vra tenant add --name NewTenant --url newtenant

  • Next add the identity store for AD authentication
vra tenant identitystore add --tenantname NewTenant --name Domain.local --url ldap://domain.local --groupbasedn 'ou=vRA,DC=domain,DC=local' --domain domain.local --userdn 'cn=adbind_vra,OU=vRA,DC=domain,DC=local' --password Password123! --type AD --userbasedn 'ou=vRA,DC=domain,DC=local'

  • Next add Tenant administrators to the new tenant
vra tenant admin update --tenantname NewTenant --role TENANT_ADMIN --action ADD --users vRA_Tenant_Admins@domain.local

  • Next add IaaS Administrators to the new tenant
vra tenant admin update --tenantname NewTenant --role IAAS_ADMIN --action ADD --users vRA_IaaS_Admins@domain.local


So enter vRA 7.x. Using CloudClient the first 2 steps are the same as before

  • Firstly login to CloudClient
vra login userpass --server vra-vip.domain.local --user administrator@vsphere.local --password Password123! --tenant vsphere.local

  • Next create the tenant
vra tenant add --name NewTenant --url newtenant

  • Next we need to create a local user in the tenant. I couldn’t find a function in CloudClient to create the local user. I checked the developer API guide and I also tried the excellent community module PowervRA but no joy so i enlisted the help of my colleague Sean Leahy @leahy_s to query the REST API. We ran the add user operation manually and monitored the process in FireFox (Press F12 to get access to the browser console debugger.) Manually enter the user details and click Ok and you will see a POST operation in the console. See below


F12 Console

  • Click on the POST operation and inspect the REST information. In the headers tab you can see the Content-Type is application/json and in the params tab you can see the post data

F12 Params

  • Using the API guide we found the required REST method to get an authentication token from vRA that will allow us to POST data. In this example i’m using Advanced REST client in Chrome to retrieve the auth token from vRA. So the URL to POST to is https://vra-appliance-FQDN/identity/api/tokens and the credentials payload needs to be JSON formatted. See below

RESt Auth to vRA

  • The response back should contain a base64 encoded string that will be used as an authorization token to post the new user.
  • So again using Advanced REST client in chrome (or your preferred method) we need to do a POST operation to create the user. This time you need to use URL https://vra-appliance-FQDN/identity/api/tenants/NewTenant/principals. In the headers you need Accept set to Application/json and Authorization set to the base64 encoded token string we got in the previous step. IMPORTANT: You must prefix the token with Bearer or it will not work! See below



  • Here is the full JSON payload as you cant see it all in the screenshot
{"@type": "User",
 "firstName": "vRA",
 "lastName": "Admin",
 "emailAddress": "vraadmin@domain.local",
 "description": "vRA Admin",
 "locked": false,
 "disabled": false,
 "password": "Password123!",
"domain": "vsphere.local",
 "principalId": {"domain": "vsphere.local",
 "name": "vraadmin"}
  • Ok deep breaths…if you’re still with me you can now proceed to adding the user to the tenant and IaaS administrator groups!
  • In the interest of trying out different tools I decided to try using PowervRA for this task. (I will revisit this using the REST API directly next week) I wont go into installing PowervRA. There are good instructions here
  • In PowervRA run the following commands
Add-vRAPrincipalToTenantRole -TenantId NewTenant -PrincipalId vraadmin@vsphere.local -RoleId CSP_TENANT_ADMIN

Add-vRAPrincipalToTenantRole -TenantId NewTenant -PrincipalId vraadmin@vsphere.local -RoleId COM_VMWARE_IAAS_IAAS_ADMINISTRATOR

Not sure why the complete difference in RoleId name format but thats the only way i could get it to work!

So now we have a tenant with a local user that is both tenant admin & IaaS admin but we still dont have an AD directory service to authenticate AD users. So for this we will again leverage the REST API. Using the same auth token as before, this time we need to hit https://vra-appliance-FQDN/identity/api/tenants/NewTenant/directories

POST Identity Store

  • Here is the full JSON payload as you cant see it all in the screenshot
@type: "IdentityStore",
domain: "domain.local",
name: "Domain.local",
alias: "Domain",
type: "AD",
userNameDn: "cn=adbind_vra,OU=EHC,DC=domain,DC=local",
password: "Password123!",
url: "ldap://domain.local:389",
groupBaseSearchDn: "ou=EHC,DC=domain,DC=local",
userBaseSearchDn: "ou=EHC,DC=domain,DC=local"

  • We can now go and add our domain users as Tenant admins and IaaS admins to allow domain users to log into the new tenant! So its a longer process than 6.x but its still doable (even if parts of it are undocumented in the API guide!). I will be working on scripting this for multiple tenants so hopefully keep an eye out for a follow up post as i fumble my way through using the vRealize Automation REST API! 🙂

Special thanks again to my colleague Sean Leahy for the REST pointers!