Managing VMs via the ESXi command line

From time to time a host may be unmanageable from vCenter / web client or you may only have console access. In my case I was bringing up a Dell EMC VxRail. During initial bringup the ESXi hosts do not get a mgmt IP if you do not have DHCP available so management with the web client is not possible. I do have iDRAC access though so can access the console. I needed to see where the VxRail manager VM was running as it comes up during an election process between the hosts. With console access it is still possible to manage VMs using esxcli.

To discover all VMs on a host run the following

  • vim-cmd vmsvc/getallvms

Once you have the output you can use the Vmid to manipulate the powerstate of a VM

  • vim-cmd vmsvc/power.get 2

In my case the VM i wanted was powered off. You can run the following to power it on

  • vim-cmd vmsvc/power.on 2

 

And there you have it. Simple VM management using vim-cmd. Explore what else you can leverage it for here

Cleanup failed requests in vRA UI

From time to time a request in vRA will fail for whatever reason. When this happens you will see the request status as failed on the requests tab. There is a greyed out delete button that for whatever reason cannot be used to delete the failed request even when logged in as a full tenant/iaas/cloud admin.

 

There are several reasons you may want to remove failed requests…maybe you may need to deliver a demo to the CIO on some new functionality and failures in the UI never look good…or maybe you just have mild OCD like me and like to cleanup any failures to restore the illusion of all being good with the world! 🙂 Whatever your reasons here is a procedure that you can use.

Disclaimer: I dont believe this procedure if fully supported by VMware so please proceed with caution.

  • SSH to your primary vRA appliance
  • Run the following to view the contents of /etc/vcac/server.xml
    • less /etc/vcac/server.xml
  • Look for the line with password= and copy everything between the “”. This password will allow you to connect to the vRA PostGres DB

  • Run the following command with the password from the above step
    • vcac-config prop-util -d –p “s2enc~K6RsAv5WGpoAt+qsnZPrKErxZ0kU1npeK/G5iMzyaWI=”
  • Next change to the postgres user
    • su postgres
  • Change to the postgres directory
    • cd /opt/vmware/vpostgres/current/bin
  • Connect to the vcac database
    • ./psql vcac -W
  • Enter the password from server.xml
  • vRA requests are store in the cat_request table. To enable us to delete a request we first need the request id. Query the cat_request table for your request ID using the requestnumber (In my case the offending failed requestnumber is 63, as seen in the first column in the screenshot above. replace with your requestnumber)
    • SELECT id,requestnumber FROM cat_request where requestnumber = ’63’;

vRA XaaS blueprint requests are referenced in 1 further table, cat_requestevent. This entriy must be deleted before you can delete the request.

  • Run the following commands to delete the request.
  • delete from cat_requestevent where request_id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;
  • delete from cat_request where id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;

The offending failed request should now be gone from the requests list in vRA!

vRA Network and Security Inventory Data Collection Failed

I’ve been playing around with Dell EMC RP4VM & vRA and needed to setup cross vCenter NSX in my lab. I’m not going to go into that setup as there are many blogs on the subject. What i will cover is an error i hit when trying to do Network and Security Inventory data collections on one of my NSX endpoints. The error from the Dem logs in vRA (Infrastructure > Monitoring > Log ) was as follows:


Workflow 'vSphereVCNSInventory' failed with the following exception:
'object' does not contain a definition for 'clusters'

After digging around for VMware KBs and blogs on the subject and coming up empty handed i went back to review my entire setup and discovered i had missed adding a vCenter cluster to the universal transport zone on the offending NSX endpoint, which is my DR site.

Once i rectified this the Network and Security Inventory Data Collection worked as expected.

Shutdown/Power up a vSAN cluster with PowerCli

I have been doing a lot of lab testing lately and using  vCloud Director is a great way to be able to run side by side tests (sometimes destructive!) against multiple environments without requiring multiple physical clusters. I wanted to emulate a Dell EMC VxRail appliance so I created a 4 node nested vSAN cluster with a vCenter appliance & external PSC running on the cluster, to use as a vCD template. When creating vCD templates it is preferable to power down the environment before adding to the vCD catalog. When it comes to powering down this environment, because the vCenter & PSC are running on the cluster, there is a chicken and egg scenario as you need to power down in this order:

  1. vCenter
  2. PSC
  3. Put ESXi hosts in maintenance mode (vSAN aware operation)
  4. Power down ESXi hosts

The inverse applies when powering up the environment.  So rather than connect to multiple different interfaces to execute the power down or power up operations I put together the following Menu based PowerCli script with the following Options

These options will call the following functions

Option 1. Shutdown vSAN Cluster & PSC/vCenter

  • ConnectViServer $vCenterFQDN $vCenterUser $vCenterPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to vCenter as we need to perform a DRS operation.
  • ChangeDRSLevel $PartiallyAutomated
    • The ChangeDRSLevel function takes an argument for the level to set it to. In this case it sets it to partially Automated to stop DRS from moving VMs around.
  • MoveVMs
    • This function will move the VMs defined in $VMList to the host defined in $VMHost. This ensures that we know where the VMs are when we come to power the cluster back up
  • ConnectViServer $VMHost $VMHostUser $VMHostPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to the host defined in the $VMHost variable
  • ShutdownVM
    • This function will gracefully shutdown the VMs in the order defined in $VMList. In this case it will shutdown the vCenter first & then the PSC. This list can be expanded to include other VMs or could be refactored to use a CSV for a large list of VMs
  • EnterMaintenanceMode
    • This function will put all hosts defined in $VMHosts into maintenance mode. As this is a vSAN cluster it passes the NoAction flag to prevent any data moment or rebuild
  • ShutdownESXiHosts
    • This function will shutdown all hosts in the cluster

Option 2. Startup vSAN Cluster & PSC/vCenter

  • ExitMaintenanceMode
    • This function will exit all hosts from maintenance mode
  • ConnectViServer $VMHost $VMHostUser $VMHostPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to the host defined in the $VMHost variable
  • StartVMs
    • This function will startup the VMs in the reverse order defined in $VMList. In this case it will startup the PSC first & then the vCenter.
  • PollvCenter
    • This function will Poll vCenter until it is up and available
  • ConnectViServer $vCenterFQDN $vCenterUser $vCenterPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to vCenter as we need to perform a DRS operation
  • ChangeDRSLevel $FullyAutomated
    • The ChangeDRSLevel function takes an argument for the level to set it to. In this case it sets it to Fully Automated as we are done with the maintenance.

The script is posted to Github here and is also posted below. This was written and tested against vSphere 6. I will update it in the coming weeks for vSphere 6.5 and hopefully leverage some of the new APIs

# Script to shutdown & startup a vSAN cluster when vCenter/PSC are running on the cluster
# Created by Brian O'Connell | Dell EMC
# Provided with zero warranty! Please test before using in anger!
# @LifeOfBrianOC
# https://lifeofbrianoc.com/

## User Variables ##
$vCenterFQDN = "vcs01.domain.local"
$vCenterUser = "VC_Admin@domain.local"
$vCenterPassword = "Password123!"
$Cluster = "MARVIN-Virtual-SAN-Cluster"
$VMList = @("VCS01", "PSC01")
$VMHosts = @("esxi04.domain.local", "esxi05.domain.local", "esxi06.domain.local", "esxi07.domain.local")
$VMHost = "esxi04.domain.local"
$VMHostUser = "root"
$VMHostPassword = "Password123!"

### DO NOT MODIFY ANYTHING BELOW THIS LINE ###

# Add Required PowerCli Modules
Get-Module -ListAvailable VM* | Import-Module

# Function to Connect to VI Host (vCenter or ESXi). Pass host, username & password to the function
Function ConnectVIServer ($VIHost, $User, $Password) {
    Write-Host " "
    Write-Host "Connecting to $VIHost..." -Foregroundcolor yellow
    Connect-VIServer $VIHost -User $User -Password $Password | Out-Null
	Write-Host "Connected to $VIHost..." -Foregroundcolor Green
    Write-Host "------------------------------------------------" -Foregroundcolor Green
}

# Define DRS Levels to stop Vms from moving away from the defined host
$PartiallyAutomated = "PartiallyAutomated"
$FullyAutomated = "FullyAutomated"

# Function to Change DRS Automation level						
Function ChangeDRSLevel ($Level) {						

    Write-Host " "
    Write-Host "Changing cluster DRS Automation Level to Partially Automated" -Foregroundcolor yellow
    Get-Cluster $cluster | Set-Cluster -DrsAutomation $Level -confirm:$false | Out-Null
    Write-Host "------------------------------------------------" -Foregroundcolor yellow
}
						
# Function to Move the Vms to a defined host so they can be easily found when starting back up
Function MoveVMs {

    Foreach ($VM in $VMList) {
        # Power down VM
        Write-Host " "
        Write-Host "Moving $VM to $VMHost" -Foregroundcolor yellow
        Get-VM $VM | Move-VM -Destination $VMHost -Confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
    }   
    Disconnect-VIServer $vCenterFQDN -confirm:$false | Out-Null
}

# Function to Shutdown VMs
Function ShutdownVM  {

    Foreach ($VM in $VMList) {
        # Power down VM
        Write-Host " "
        Write-Host "Shutting down $VM" -Foregroundcolor yellow
        Shutdown-VMGuest $VM -Confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "Waiting for $VM to be Shutdown" -Foregroundcolor yellow
        # Check VM powerstate and wait until it is powered off before proceeding with the next VM
        do {
            sleep 15
            $powerState = (get-vm $VM).PowerState
        }
        while ($powerState -eq "PoweredOn")
        Write-Host " "
        Write-Host "$VM Shutdown.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
}

# Function to put all ESXi hosts into maintenance mode with the No Action flag for vSAN data rebuilds
Function EnterMaintenanceMode {

    Foreach ($VMHost in $VMHosts) {
        Connect-VIServer $VMHost -User root -Password $VMHostPassword | Out-Null
        # Put Host into Maintenance Mode
        Write-Host " "
        Write-Host "Putting $VMHost into Maintenance Mode" -Foregroundcolor yellow
        Get-View -ViewType HostSystem -Filter @{"Name" = $VMHost }|?{!$_.Runtime.InMaintenanceMode}|%{$_.EnterMaintenanceMode(0, $false, (new-object VMware.Vim.HostMaintenanceSpec -Property @{vsanMode=(new-object VMware.Vim.VsanHostDecommissionMode -Property @{objectAction=[VMware.Vim.VsanHostDecommissionModeObjectAction]::NoAction})}))}
        Disconnect-VIServer $VMHost -confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "$VMHost in maintenance mode.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
}

# Function to Exit hosts from maintenance mode
Function ExitMaintenanceMode {

    Foreach ($VMHost in $VMHosts) {
        Connect-VIServer $VMHost -User root -Password $VMHostPassword | Out-Null
        # Exit Maintenance Mode
        Write-Host " "
        Write-Host "Exiting Maintenance Mode for $VMHost" -Foregroundcolor yellow
        Set-VMHost $VMHost -State "Connected" | Out-Null
        Disconnect-VIServer $VMHost -confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "$VMHost out of maintenance mode.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
    Write-Host "Waiting for vSAN Cluster to be Online" -Foregroundcolor yellow	
	Sleep 60							
}

# Function to shutdown hosts
Function ShutdownESXiHosts {

    Foreach ($VMHost in $VMHosts) {
        # Exit Maintenance Mode
        Write-Host " "
        Write-Host "Shutting down ESXi Hosts" -Foregroundcolor yellow
        Connect-VIServer -Server $VMHost -User root -Password $VMHostPassword | %{
            Get-VMHost -Server $_ | %{
                $_.ExtensionData.ShutdownHost_Task($TRUE) | Out-Null
            }
        }
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "ESXi host $VMHost shutdown.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
		Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "All ESXi Hosts shutdown.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
}

# Function to Start VMs in the reverse order they were powered down									
Function StartVMs {
    # Reverse the VM list to start in reverse order
    [array]::Reverse($VMList)

    Foreach ($VM in $VMList) {
        # Power on VM
        Write-Host " "
        Write-Host "Powering on $VM" -Foregroundcolor yellow
        Start-VM $VM -Confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "Waiting for $VM to be Powered On" -Foregroundcolor yellow
        # Check VM powerstate and wait until it is powered on before proceeding with the next VM
        do {
            sleep 15
            $powerState = (get-vm $VM).PowerState
        }
        while ($VM -eq "PoweredOff")
        Write-Host " "
        Write-Host "$VM Powered On..proceeding with next VM" -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }

}

# Function to Poll the status of vCenter after starting up the VM
Function PollvCenter {

    do 
    {
        try 
        {
            Write-Host " "
            Write-Host "Polling vCenter $vCenterFQDN Availability...." -ForegroundColor Yellow
            Write-Host "------------------------------------------------" -Foregroundcolor yellow
            # Create Web Request
            [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
            $HTTP_Request = [System.Net.WebRequest]::Create("https://$($vCenterFQDN):9443")

            # Get a response
            $HTTP_Response = $HTTP_Request.GetResponse()

            # Get the HTTP code
            $HTTP_Status = [int]$HTTP_Response.StatusCode

            If ($HTTP_Status -eq 200) { 
                Write-Host " "
                Write-Host "vCenter $vCenterFQDN is Available!"  -ForegroundColor Green
                Write-Host "------------------------------------------------" -Foregroundcolor Green
                # Close HTTP request
                $HTTP_Response.Close()
            }
        }
        catch { 
            Write-Host " "
            Write-Host "vCenter $vCenterFQDN Not Available Yet...Retrying Poll..."  -ForegroundColor Cyan
            Write-Host "------------------------------------------------" -Foregroundcolor Cyan
    } }
    While ($HTTP_Status -ne 200)	
}

# Function to display the main menu 
Function Menu 
{
    Clear-Host         
    Do
    {
        Clear-Host                                                                        
        Write-Host -Object 'Please choose an option'
        Write-Host     -Object '**********************'	
        Write-Host -Object 'vCenter & vSAN Maintenance Options' -Foregroundcolor Yellow
        Write-Host     -Object '**********************'
        Write-Host -Object '1.  Shutdown vSAN Cluster & PSC/vCenter '
        Write-Host -Object ''
        Write-Host -Object '2.  Startup vSAN Cluster & PSC/vCenter '
        Write-Host -Object ''
        Write-Host -Object 'Q.  Exit'
        Write-Host -Object $errout
        $Menu = Read-Host -Prompt '(Enter 1 - 2 or Q to quit)'

        switch ($Menu) 
        {
            1 
            {
				ConnectVIServer $vCenterFQDN $vCenterUser $vCenterPassword
                ChangeDRSLevel $PartiallyAutomated
                MoveVMs
                ConnectVIServer $VMHost $VMHostUser $VMHostPassword
                ShutdownVM
                EnterMaintenanceMode
                ShutdownESXiHosts   
            }
            2 
            { 
				ExitMaintenanceMode
                ConnectVIServer $VMHost $VMHostUser $VMHostPassword
                StartVMs
				PollvCenter
                ConnectVIServer $vCenterFQDN $vCenterUser $vCenterPassword
                ChangeDRSLevel $FullyAutomated
            }
            Q 
            {
                Exit
            }	
            default 
            {
                $errout = 'Invalid option please try again........Try 1-2 or Q only'
            }

        }
    }
    until ($Menu -eq 'q')
}   

# Launch The Menu
Menu

Setup multiple vRA tenants using powershell and the vRA7 REST API

Following on from my post on Creating a local user in vRA7 using the REST API i wanted to try and script the entire process of creating multiple vRealize Automation 7 tenants as in our lab as we often need to spin up multiple tenants for testing or development purposes.

Some assumptions:

  • Each tenant has the same prefix of “dev-” and is appended with a 3 digit number starting at “001”
  • Each tenant gets the same local user created with matching credentials
  • Each tenant gets the same AD directory added
  • Each tenant gets the same AD groups added

This script will do the following:

  • Log into the default tenant
  • Create a new tenant
  • Create a local user for the tenant
  • Add the local user as a tenant & IaaS admin
  • Log into the new tenant as the local user
  • Setup identity store directories
  • Log back into the default tenant
  • Edit the new tenant
  • Add domain users/groups as tenant & IaaS admins
  • Log into the new tenant as a tenant and IaaS admin and start configuring the tenant

So as to avoid the requirement to edit the powershell script directly i put all configuration variables in an external .cfg file. This file needs to be placed in the same directory as the powershell script.

Firstly here is the config file contents. Edit each variable to match your environment. Modify the numberOfTenants variable to set the number of tenants you want to create. The example below will create 20 tenants.

[vRA FQDN]
VRA=vra-vip.domain.local

[vRA Credentials to acquire authentication token]
vRAUsername=administrator@vsphere.local
vRAPassword=Password123!
vRADefaultTenant=vsphere.local

[Create tenant details]
numberOfTenants=2
tenantIDPrefix=dev-
tenantURLPrefix=dev-
tenantNamePrefix=dev-
tenantDescription=DevelopmentTenant
tenantemailAddress=admin@vsphere.local

[Local Admin User Details]
firstName=vRA
lastName=Admin
emailAddress=vraadmin@vsphere.local
description=vRAAdmin
locked=false
disabled=false
password=Password123!
domain=vsphere.local
userName=vraadmin
name=vraadmin

[Tenant Directory Details]
adDomain=domain.local
adDomainalias=Domain
type=AD
adUserNameDn=cn=adbind_vra,OU=EHC,DC=domain,DC=local
adBindPassword=Password123!
adURL=ldap://domain.local:389
adGroupBaseSearchDn=ou=EHC,DC=domain,DC=local
adUserBaseSearchDn=ou=EHC,DC=domain,DC=local

[AD Domain Groups to add as Tenant & IaaS Admins]
tenantAdmins=EHC_Tenant_Admins@domain.local
tenantRoleID=CSP_TENANT_ADMIN
iaasAdmins=EHC_IaaS_Admins@domain.local
iaasRoleID=COM_VMWARE_IAAS_IAAS_ADMINISTRATOR

And here is the script to create the tenants. It is broken up into multiple functions

# Script to create vRA7 Tenants in bulk
# Ensure you update the associated cfg file
# with the details of your vRA environment
# and details of the tenants you wish to create
# Created by Brian O'Connell
# Version 1.0.0

# Import configuration variables from external cfg file
Get-Content createvRATenants.cfg | Foreach-Object{
if ($_.length -gt 0) {
 $var = $_ -Split '=',2
 New-Variable -Name $var[0] -Value $var[1]
 }
 } 

Function getvRAAuthToken {
# Construct credentials from config file
$credentials=@{username=$vRAUsername;password=$vRAPassword;tenant=$vRADefaultTenant}
############# Get Auth token ###############
$headers=@{
 "Accept"="application/json"
}
$Global:token = Invoke-RestMethod -Uri "https://$($VRA)/identity/api/tokens" -Method Post -Headers $headers -ContentType application/json -Body (ConvertTo-Json $credentials) | Select -ExpandProperty id
Write-Host "vRA Authentication Token Acquired" -ForegroundColor Green
 } 

Function createvRATenant {
 # ############ Create Tenant ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Create the Tenant
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
$tenantid = -join ($tenantIDPrefix,$tenantNumber)
$tenantURL = -join ($tenantURLPrefix,$tenantNumber)
$tenantName = -join ($tenantNamePrefix,$tenantNumber)
$tenantBody= @"
{
 "@type": "Tenant",
 "id": "$tenantid",
 "urlName": "$tenantURL",
 "name": "$tenantName",
 "description": "$tenantDescription",
 "contactEmail": "$tenantemailAddress"
}
"@ 

$createTenant = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/tenants/$($tenantID)" -headers $headers -ContentType application/json -body $tenantBody
Write-Host "Tenant $($tenantName) created successfully" -ForegroundColor Green
}
 }

Function createvRALocalAdminUser {
 ############# Create Local Admin User ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$userBody= @"
{ "@type": "User",
 "firstName": "$firstName",
 "lastName": "$lastName",
 "emailAddress": "$emailAddress",
 "description": "$description",
 "locked": false,
 "disabled": false,
 "password": "$password",
 "domain": "$domain",
 "userName": "$userName",
 "principalId": {
 "domain": "$domain",
 "name": "$name"
 }
}
"@

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the user
$createUser = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/principals" -headers $headers -ContentType "application/json" -body $userBody
Write-Host "Local Admin User for tenant $($tenantid) created successfully" -ForegroundColor Green
} 

 }

Function updatevRALocalAdminUserRoles {
 ############# Add Local Admin User to Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")
$principal = "vraadmin@vsphere.local"
$roleIDs = @("CSP_TENANT_ADMIN","COM_VMWARE_IAAS_IAAS_ADMINISTRATOR")

for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Add the user to tenant & IaaS admins
foreach ($roleID in $roleIDs) {
$makeUserAdmin = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($principal)/roles/$($roleID)" -headers $headers -body "{}"
}
Write-Host "Local Admin User Added to Tenant & IaaS Admins for tenant $($tenantid) " -ForegroundColor Green
 }
 }

Function createvRATenantDirectory {
 ############# Add AD Tenant directory ###############
$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

$directoryBody= @"
{"@type": "IdentityStore",
"domain": "$adDomain",
"name": "$adDomain",
"alias": "$adDomainalias",
"type": "$type",
"userNameDn": "$adUserNameDn",
"password": "$adBindPassword",
"url": "$adURL",
"groupBaseSearchDn": "$adGroupBaseSearchDn",
"userBaseSearchDn": "$adUserBaseSearchDn"
}
"@
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
#Create the directory
$createDirectory = Invoke-RestMethod -Method Post -URI "https://$($VRA)/identity/api/tenants/$($tenantID)/directories" -headers $headers -ContentType "application/json" -body $directoryBody
Write-Host "Tenant Directory Created for tenant $($tenantid) " -ForegroundColor Green
}

 }

Function addDomainGroupstovRAAdmins {
############## Add AD Domain Groups to vRA Tenant & IaaS Admin groups ###############

$headers = @{"Accept" = "application/json"}
$headers.Add("Authorization", "Bearer $token")

#Add the user to tenant & IaaS admins
for ($firstTenantNumber=1; $firstTenantNumber -le $numberOfTenants; $firstTenantNumber++)
{
 New-Variable -Name "var$firstTenantNumber" -Value $firstTenantNumber
 $tenantNumber = $firstTenantNumber.ToString("000")
 $tenantid = -join ($tenantIDPrefix,$tenantNumber)
$addTenantAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($tenantAdmins)/roles/$($tenantRoleID)" -headers $headers -body "{}"

$addIaaSAdmins = Invoke-RestMethod -Method PUT -URI "https://$($VRA)/identity/api/authorization/tenants/$($tenantID)/principals/$($iaasAdmins)/roles/$($iaasRoleID)" -headers $headers -body "{}"
Write-Host "Domain groups added to as tenant & IaaS admins for tenant $($tenantid) " -ForegroundColor Green
}
 }

# Call All functions to setup tenants
getvRAAuthToken; createvRATenant; createvRALocalAdminUser; updatevRALocalAdminUserRoles; createvRATenantDirectory; addDomainGroupstovRAAdmins 

Setting up a VMware Appliance update repo

When upgrading VMware appliances using the vami configuration interface the update that is available from VMware does not always match the version you want. mounting an ISO to the VM is not always possible (vCloud Director hosted VMs are an example). Follow this procedure to create a temporary repo to make the correct upgrade bundles available during appliance upgrade
In this example the vRA appliance is used to host the repo. Any VMware appliance can be used for this purpose.
1. SCP the upgrade ISO to /tmp on the vRA appliance
2. SSH to the vRA appliance and navigate to the tmp directory

 cd /tmp

3. Create a new directory called repo

Mkdir repo
 4. Type the following to mount the upgrade ISO to the new repo directory
mount -o loop "iso name" repo
 5. Change directory into repo/update
 cd repo/update
 6. Start the python SimpleHTTPServer
 python -m SimpleHTTPServer
 7. Return to the VAMI configuration interface of the appliance that you are upgrading and browse to the Update tab
 8. Click Settings
 9. Change the Update Repository option to Use Specified Repository
 10. For Repository URL enter http://vRA-FQDN:8000
 Note: Enter the FQDN of the appliance you are using. This may not be the vRA appliance
 11. Click Save Settings
 12. Click Status and click Check Updates
 13. The correct version that you copied to the temporary repo should be available to install

Distributed vRA validation script

From time to time a distributed vRA deployment can have issues…here is a quick script to verify and validate the important components are up and functioning…without the need to log into multiple components. Here is a diagram of my distributed vRA setup

Distributed vRA v4

And here is the script! It has multiple functions to do the following

  • Basic ping tests to each component
  • Get the status of all vRA component services
  • Test the Web & manager server URLs

# Script to check the status of each component of a distributed vRA deployment
# Modify the hostnames to match your environment
$vRAAppliance01FQDN = "vra01.domain.local"
$vRAAppliance02FQDN = "vra02.domain.local"
$vRAWeb01FQDN = "web01.domain.local"
$vRAWeb02FQDN = "web02.domain.local"
$vRAManager01FQDN = "manager01.domain.local"
$vRAManager02FQDN = "manager02.domain.local"
$vRADEM01FQDN = "demw01.domain.local"
$vRADEM02FQDN = "demw02.domain.local"
$vRAAgent01FQDN = "agent01.domain.local"
$vRAAgent02FQDN = "agent02.domain.local"
$vRAComponentServiceURL = "https://vra-vip.domain.local/component-registry/services/status/current"
$webVIPURL = "https://web-vip.domain.local/WAPI"
$managerVIPURL = "https://manager-vip.domain.local/VMPS2"



### DO NOT MODIFY ANYTHING BELOW THIS LINE ###



Function pingHosts {
@"
===============================================================================
Performing basic ping test to each defined component
===============================================================================
"@
$vms = @($vRAAppliance01FQDN, $vRAAppliance02FQDN, $vRAWeb01FQDN, $vRAWeb02FQDN, $vRAManager01FQDN, $vRAManager02FQDN, $vRADEM01FQDN, $vRADEM02FQDN, $vRAAgent01FQDN, $vRAAgent02FQDN)
$collection = $()
foreach ($vm in $vms)
{
 $status = @{ "ServerName" = $vm; "TimeStamp" = (Get-Date -f s) }
 if (Test-Connection $vm -Count 1 -ea 0 -Quiet)
 { 
 $status["Ping Results"] = "Up"
 } 
 else 
 { 
 $status["Ping Results"] = "Down" 
 }
 New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
 $collection += $serverStatus

}
 }
 
Function testvRAServices {
@"
===============================================================================
Getting Status of all vRA component services
===============================================================================
"@
Write-Host "Checking status of vRA Component Services" -ForegroundColor Yellow
# Request Service Information from $vRAComponentServiceURL
$vRAURL = Invoke-WebRequest $vRAComponentServiceURL
# Convert the Json response
$json = ConvertFrom-Json -InputObject $vRAURL.content
# Get Service name & status
$serviceInfo = $json.content
# Loop through each service
foreach ($service in $serviceInfo) {
# Get the Service Name
$serviceName = $service.serviceName
# Get the Service status
$serviceStatus = $service.serviceStatus.serviceInitializationStatus
# If Service Status is blank report it as BLANK POSSIBLY STOPPED
 if (!$serviceStatus) {
 $serviceStatus = "BLANK - POSSIBLY STOPPED?"
 }
# If Service Status is FAILED print to screen in red 
if ($serviceStatus -eq "FAILED") {
 write-host "$serviceName $serviceStatus" -ForeGroundColor Red
 }
# Otherwise print to screen as normal (Remove this if you only want to report failed services) 
 else {
 Write-Host "$serviceName $serviceStatus"}
 }

}


Function testWebVIP {
@"
===============================================================================
Checking status of vRA Web API URL
===============================================================================
"@
Write-Host "Testing IaaS Web Service VIP URL $webVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($webVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Web Service is OK!" -ForegroundColor Green
 # Close HTTP request
$HTTP_Response.Close()
}
Else {
 Write-Host "IaaS Web Service is not responding. Restart IIS on Web01. If that does not resolve then Reboot Web01" -ForegroundColor Red
}
 }
 
Function testMgrVIP {
@"
===============================================================================
Checking status of vRA Manager API URL
===============================================================================
"@
Write-Host "Testing IaaS Manager Service VIP URL $managerVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($managerVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Manager Service is OK!" -ForegroundColor Green
 # Close HTTP request
$HTTP_Response.Close()
}
Else {
 Write-Host "IaaS Manager Service is not responding. Ensure all vRA services are running on manager01. If that does not resolve then Reboot manager01" -ForegroundColor Red
}
 } 
 

 pingHosts; testvRAServices; testWebVIP; testMgrVIP

The function pingHosts is a basic ping test to each defined vRA component

The function testvRAServices was an interesting one to write as I’m not overly familiar with working with APIs so it was a learning experience. I wanted to be able to report the status of all vRA services listed on the VAMI administration UI of a vRA appliance (https://vra:5480). The URL that the services are listed on is https://vra-vip.domain.local/component-registry/services/status/current so using the powershell Invoke-WebRequest you get back the page information.

Invoke-WebRequest

Line 56 in the script puts the page contents into a variable we can work with. You can see that the information we want is stored in Content (ServiceStatus) in Json format so you need to take that Json and convert it to  to powershell readable text using the ConvertFrom-Json function (ConvertFrom-Json converts a JSON-formatted string to a custom object (PSCustomObject) that has a property for each field in the JSON string) Line 58 does this

We then put each service into the $serviceinfo variable and loop through them to get the service name and service status.

Side note here: Originally I was querying $json.content.serviceStatus to get the details i wanted but i noticed I wasnt getting a full list of services, i was getting some blank content and also some duplicate service names. This is how i was doing it

$vRAURL = Invoke-WebRequest "https://vra-vip.vlab.local/component-registry/services/status/current" 
$json = ConvertFrom-Json -InputObject $vRAURL.content 
$serviceInfo = $json.content.serviceStatus | Select serviceName,serviceInitializationStatus $serviceInfo 

Here is that that returns. As you can see its not the full list and there is a duplicate entry so its not much use

Duplicate Results

I dug a little into the API and it seems that it does indeed contain inconsistent information. Here is an excerpt with some issues where the actual service name is content-management but the serviceStatus reports the name as advanced-designer-service

Service Name issue

So to get an accurate list i queried the serviceName field to get the name and the serviceStatus.serviceInitializationStatus to get the service status. Unfortunately doing it this way doesnt allow creating a nice formatted table (at least i havent figured out how to do it yet!) but i did get it to print out each service & status on the same line.

Line 68: In my lab i use a vRO appliance so the internal vRO service on the vRA appliance is stopped. The service status comes back blank so i added a check to report blank service status as “BLANK – POSSIBLY STOPPED?”.

Line 72: I also added a check to print any failed services in red so they stand out.

The testWebVIP and testManagerVIP functions use the powershell System.Net.WebRequest to get the HTTP status code for a given URL. If the status code comes back as 200 then everything is ok. If not there is an issue with your IaaS components

So there you have it. A quick way to verify the status of all of the important vRealize Automation components and services. In my example below the iaas-service is in a failed state (The driving reason for creating this script! 🙂 )

Script Results