Distributed vRA validation script

From time to time a distributed vRA deployment can have issues…here is a quick script to verify and validate the important components are up and functioning…without the need to log into multiple components. Here is a diagram of my distributed vRA setup

Distributed vRA v4

And here is the script! It has multiple functions to do the following

  • Basic ping tests to each component
  • Get the status of all vRA component services
  • Test the Web & manager server URLs

# Script to check the status of each component of a distributed vRA deployment
# Modify the hostnames to match your environment
$vRAAppliance01FQDN = "vra01.domain.local"
$vRAAppliance02FQDN = "vra02.domain.local"
$vRAWeb01FQDN = "web01.domain.local"
$vRAWeb02FQDN = "web02.domain.local"
$vRAManager01FQDN = "manager01.domain.local"
$vRAManager02FQDN = "manager02.domain.local"
$vRADEM01FQDN = "demw01.domain.local"
$vRADEM02FQDN = "demw02.domain.local"
$vRAAgent01FQDN = "agent01.domain.local"
$vRAAgent02FQDN = "agent02.domain.local"
$vRAComponentServiceURL = "https://vra-vip.domain.local/component-registry/services/status/current"
$webVIPURL = "https://web-vip.domain.local/WAPI"
$managerVIPURL = "https://manager-vip.domain.local/VMPS2"



### DO NOT MODIFY ANYTHING BELOW THIS LINE ###



Function pingHosts {
@"
===============================================================================
Performing basic ping test to each defined component
===============================================================================
"@
$vms = @($vRAAppliance01FQDN, $vRAAppliance02FQDN, $vRAWeb01FQDN, $vRAWeb02FQDN, $vRAManager01FQDN, $vRAManager02FQDN, $vRADEM01FQDN, $vRADEM02FQDN, $vRAAgent01FQDN, $vRAAgent02FQDN)
$collection = $()
foreach ($vm in $vms)
{
 $status = @{ "ServerName" = $vm; "TimeStamp" = (Get-Date -f s) }
 if (Test-Connection $vm -Count 1 -ea 0 -Quiet)
 { 
 $status["Ping Results"] = "Up"
 } 
 else 
 { 
 $status["Ping Results"] = "Down" 
 }
 New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
 $collection += $serverStatus

}
 }
 
Function testvRAServices {
@"
===============================================================================
Getting Status of all vRA component services
===============================================================================
"@
Write-Host "Checking status of vRA Component Services" -ForegroundColor Yellow
# Request Service Information from $vRAComponentServiceURL
$vRAURL = Invoke-WebRequest $vRAComponentServiceURL
# Convert the Json response
$json = ConvertFrom-Json -InputObject $vRAURL.content
# Get Service name & status
$serviceInfo = $json.content
# Loop through each service
foreach ($service in $serviceInfo) {
# Get the Service Name
$serviceName = $service.serviceName
# Get the Service status
$serviceStatus = $service.serviceStatus.serviceInitializationStatus
# If Service Status is blank report it as BLANK POSSIBLY STOPPED
 if (!$serviceStatus) {
 $serviceStatus = "BLANK - POSSIBLY STOPPED?"
 }
# If Service Status is FAILED print to screen in red 
if ($serviceStatus -eq "FAILED") {
 write-host "$serviceName $serviceStatus" -ForeGroundColor Red
 }
# Otherwise print to screen as normal (Remove this if you only want to report failed services) 
 else {
 Write-Host "$serviceName $serviceStatus"}
 }

}


Function testWebVIP {
@"
===============================================================================
Checking status of vRA Web API URL
===============================================================================
"@
Write-Host "Testing IaaS Web Service VIP URL $webVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($webVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Web Service is OK!" -ForegroundColor Green
 # Close HTTP request
$HTTP_Response.Close()
}
Else {
 Write-Host "IaaS Web Service is not responding. Restart IIS on Web01. If that does not resolve then Reboot Web01" -ForegroundColor Red
}
 }
 
Function testMgrVIP {
@"
===============================================================================
Checking status of vRA Manager API URL
===============================================================================
"@
Write-Host "Testing IaaS Manager Service VIP URL $managerVIPURL" -ForegroundColor Yellow
# Create Web Request
$HTTP_Request = [System.Net.WebRequest]::Create($managerVIPURL)

# Get a response
$HTTP_Response = $HTTP_Request.GetResponse()

# Get the HTTP code
$HTTP_Status = [int]$HTTP_Response.StatusCode

If ($HTTP_Status -eq 200) { 
 Write-Host "IaaS Manager Service is OK!" -ForegroundColor Green
 # Close HTTP request
$HTTP_Response.Close()
}
Else {
 Write-Host "IaaS Manager Service is not responding. Ensure all vRA services are running on manager01. If that does not resolve then Reboot manager01" -ForegroundColor Red
}
 } 
 

 pingHosts; testvRAServices; testWebVIP; testMgrVIP

The function pingHosts is a basic ping test to each defined vRA component

The function testvRAServices was an interesting one to write as I’m not overly familiar with working with APIs so it was a learning experience. I wanted to be able to report the status of all vRA services listed on the VAMI administration UI of a vRA appliance (https://vra:5480). The URL that the services are listed on is https://vra-vip.domain.local/component-registry/services/status/current so using the powershell Invoke-WebRequest you get back the page information.

Invoke-WebRequest

Line 56 in the script puts the page contents into a variable we can work with. You can see that the information we want is stored in Content (ServiceStatus) in Json format so you need to take that Json and convert it to  to powershell readable text using the ConvertFrom-Json function (ConvertFrom-Json converts a JSON-formatted string to a custom object (PSCustomObject) that has a property for each field in the JSON string) Line 58 does this

We then put each service into the $serviceinfo variable and loop through them to get the service name and service status.

Side note here: Originally I was querying $json.content.serviceStatus to get the details i wanted but i noticed I wasnt getting a full list of services, i was getting some blank content and also some duplicate service names. This is how i was doing it

$vRAURL = Invoke-WebRequest "https://vra-vip.vlab.local/component-registry/services/status/current" 
$json = ConvertFrom-Json -InputObject $vRAURL.content 
$serviceInfo = $json.content.serviceStatus | Select serviceName,serviceInitializationStatus $serviceInfo 

Here is that that returns. As you can see its not the full list and there is a duplicate entry so its not much use

Duplicate Results

I dug a little into the API and it seems that it does indeed contain inconsistent information. Here is an excerpt with some issues where the actual service name is content-management but the serviceStatus reports the name as advanced-designer-service

Service Name issue

So to get an accurate list i queried the serviceName field to get the name and the serviceStatus.serviceInitializationStatus to get the service status. Unfortunately doing it this way doesnt allow creating a nice formatted table (at least i havent figured out how to do it yet!) but i did get it to print out each service & status on the same line.

Line 68: In my lab i use a vRO appliance so the internal vRO service on the vRA appliance is stopped. The service status comes back blank so i added a check to report blank service status as “BLANK – POSSIBLY STOPPED?”.

Line 72: I also added a check to print any failed services in red so they stand out.

The testWebVIP and testManagerVIP functions use the powershell System.Net.WebRequest to get the HTTP status code for a given URL. If the status code comes back as 200 then everything is ok. If not there is an issue with your IaaS components

So there you have it. A quick way to verify the status of all of the important vRealize Automation components and services. In my example below the iaas-service is in a failed state (The driving reason for creating this script! 🙂 )

Script Results

vRA 6.2 Distributed IaaS deployment

I needed to deploy a distributed vRA 6.2 IaaS in the lab and didn’t find too many resources on the web so here are some of my notes! I hope to revisit this soon to do a full distributed HA deployment including NSX load balancers. For now here is a standard distributed deployment

Pre-Reqs/Assumptions

  • AD installed & available
  • vCenter/SSO installed & available
  • SQL Server installed and available and IaaS SQL pre reqs run for MSDTC
  • vRA 6.2 appliance deployed and integrated with SSO
  • AD CA configured (Will try to cover this in another post)
  • 5 Windows 2012 R2 VMs. 1 for each of the following roles – Web Server, Manager Server, DEM Orchestrator, DEM Worker, IaaS Agent
  • All windows VMs joined the AD domain
  • Run Brian Graf’s IaaS pre req script on all VMs (Pretty sure its only required on the Web & manager server but didnt take any chances!)

Here are the steps

  • Download the IaaS installer from the vRA appliance @ https://vra-FQDN:5480/i
  • First we need to install the database component. From any VM run the IaaS installer
  • Accept the ELUA and click Next
  • Enter the root credentials for the vRA appliance and select Accept Certificate and click Next
  • Choose Custom Install > IaaS Server

Installation Type

  • Select Database from the features list and enter the database instance and database name and click Next (I’m using SQL Express in the lab)

2.install DB

  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

3.DB Pre Reqs

  • Click Install on the Ready to Install Screen and Next and Finish once the Database install completes
  • Next you need to install the Website feature
  • Before installing the website component it is recommended to generate a CA signed cert for the server FQDN.
  • To do this open IIS on the Web server, select your server name and select Server Certificates
  • In the right hand pane select Create Domain Certificate

Generate Certificate

  • Enter the certificate distinguished name for the web server and click Next

Certificate Request

 

  • Click Select to choose your CA and assign a friendly name (user the web server FQDN)
  • Once complete run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Website from the feature list.
  • Choose the Default Web Site and the default port of 443
  • Under Available certificates choose the CA signed certificate you created earlier and enter the web server FQDN for the IaaS Server

web server setup

  • Because the IaaS Model Manager Data is required for the Website component to run you must also select the ModelManagerData feature
  • Enter the vCAC/vRA appliance FQDN & SSO details and also the IaaS Web Server FQDN and click Next

9. ModelManagerData

 

  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

10. ModelManagerPreReqs

  • On the Server and Account Settings screen enter the password for the service account being used and a passphrase for the database and click Next and click Finish once the install completes

11. ModelManagerData Account settings

  • Next up is the Manager Service
  • On the VM you designated for the Manager Service follow the steps outlined earlier to generate a CA signed certificate
  • Once complete run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Manager Service from the feature list.
  • Enter the FQDN of the Web Server and select the certificate you created in the previous steps and click Next and click Finish

managerService Install

 

  • Next up is the DEM Orchestrator
  • On the VM you designated for the DEMO Orchestrator run the IaaS installer, choose custom > Distributed Execution Managers and click Next
  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Distributed Execution Manager Screen do the following
    • From the DEM role drop down choose Orchestrator
    • Enter a name
    • Enter a description
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Click Add and click Next and click Install and then click Finish

DEM Orchestratot Config

 

  • Repeat the above steps on the DEM Worker VM choosing the DEM Worker Role
  • Finally we will install a Proxy Agent that will be used to communicate with vRA endpoints
  • On the VM you designated for the IaaS Agent run the IaaS installer, choose custom > Proxy Agents and click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Proxy Agent Screen do the following
    • From the Agent Type drop down choose vSphere
    • Enter a name
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Enter the vCenter Endpoint FQDN
    • Click Add and click Next and click Install and then click Finish

Proxy Agent Install

  • If everything went according to plan you should now be able to log into vRA and configure tenants and resources and start using your distributed IaaS installation.

Hopefully if you’ve read this far you found this useful. I hope to post the distributed HA procedure soon once i get some free lab time!