Reload all invalid virtual machines on a single host

From time to time VMs can show in an invalid state in vCenter. It can be difficult to get them back to a normal state. This is a useful command that i found at the end of this KB https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003742

for a in $(vim-cmd vmsvc/getallvms 2>&1 |grep invalid |awk '{print $4}'|cut -d \' -f2);do vim-cmd vmsvc/reload $a;done

If you get an Unexpected token error when running the above command it is probaby due to wordpress messing up the ‘ characters so either replace them or copy the command from the KB!

Failed vRA IaaS Web Server Install

There are many reasons why a vRealize Automation IaaS Web server install can fail

  • MSDTC Issues
  • NTP Issues
  • DNS Issues
  • Certificate Issues
  • Mental Issues…. (Caused by all of the above!)

I hit this error in the lab and the fix was a new one to me so said i’d document it. The IaaS web server was failing to install and the error in the logs is below which pointed to the certs

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

I checked and double checked all certificates and SAN attributes of each component, checked DNS resolution forward and reverse, verified NTP etc. Then it occurred to me that the signing CA is from a different domain than the domain i was installing in so i added the root cert to the local trusted root certification store and low and behold the install completed successfully. Had to do the same on all other IaaS VMs. So even though the root and signing certs were in the personal store with the SAN certs it was not sufficient.

 

Quick linux tip: Find all listening ports

This is probably widely known but every time i need to do it i need to search for the commands so putting this here so i can easily find it the next time i need it! 🙂

netstat -pnlt | sort

netstat

 

Find MoRef ID using powercli

From time to time you need to find the moref of an object in vCenter. This is a quick powercli one liner to get the name and ID of an object type (VM, Datastore etc)

This assumes you are already connected to vCenter using Connect-VIServer

# Datastores

Get-Datastore | Select Name,ID

This command returns output like below. In this example i have 1 datastore

Get-Datastore

#VMs

Get-VM | Select Name,ID

This command returns output like below

Get-VM

Use PowerCLI to patch multiple hosts

I was chatting to a friend who was looking to patch multiple (120) hosts with the same VIB and we discussed using PowerCLI to automate it. I did a quick google and didn’t find a script to do exactly what we needed. I did however find a script to do most of what we needed and modified it to do the VIB install and some basic logging! The script i used as a baseline is posted here on http://www.virtadmin.com/ Below is my modified version to install a VIB on each host before the reboot. ***As with any script please test extensively before running in a production environment.***

The full script can be found here

##############################################
# Script to patch multiple hosts with a VIB
# Inspired by http://www.virtadmin.com/rolling-reboot-vsphere-cluster-powercli/
# Hosts are listed in a text file
# Requires Powercli
##############################################
# Region User Variables

# Set vCenter Hostname Variable. 
# You will be asked for credentials when executing the script. e.g. "vc01.lab.local"
$vCenterServer = "ChangeME"

# Full Path to the text file with the list of ESXi hosts to be patched. 
# e.g. "C:\Scripts\VIHosts.txt"
$VIHosts = "ChangeME"

# Full path to the VIB to be installed. Use a common shared datastore. 
# Suggest a shared NFS/VMFS datastore. Copy the VIB to this location before starting. 
# e.g. "/vmfs/volumes/NFS_Shared/patch1/metadata.zip"
$vibPath = "ChangeME"

# Full path to where you would like the script to create a log. e.g. "C:\Scripts"
$LogDIR = "ChangeME"

# End Region User Variables

# DO NOT MODIFY BELOW THIS LINE!
##############################################

# Load VMware PowerCLI CmdLets
Add-PSSnapin vmware* | Out-Null

$TargetDIR = "$LogDIR\log"
if(!(Test-Path -Path $TargetDIR )){
    New-Item -ItemType directory -Path $TargetDIR
}

$Logfile = "$TargetDIR\patchESXi-Log.txt"
if(!(Test-Path -Path $Logfile )){
    New-Item -ItemType file -Path $Logfile
}

Function verifyTXTPath {
# Verify TXT Path
 if (!(Test-Path $VIHosts))  {
  Write-Host "ESXi Hosts File Not Found. Please verify the path and retry
						 " -ForegroundColor Red
	Write-Host -NoNewLine 'Press any key to exit...' -ForegroundColor Yellow;
$null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown');
Exit
		}
		  }


# Function to write Logfile entries
Function LogWrite
{
   Param ([string]$logstring)

   Add-content $Logfile -value $logstring
   }

   
# Connect to  the vCenter defined at the top of this script
Connect-VIServer -Server $vCenterServer | Out-Null

# Write progress to LogFile
LogWrite "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") Connected to vCenter $vCenterServer"
 
# Get ESXi hostname from txt file
$ESXiServers = Get-Content $VIHosts | %{Get-VMHost $_}
 
# Install ESXi Patch Function
Function PatchESXiServer ($CurrentServer) {
    # Get ESXi Server name
    $ServerName = $CurrentServer.Name
 
    # Put server in maintenance mode
    Write-Host "** Patching $ServerName **"
    Write-Host "Entering Maintenance Mode"
    Set-VMhost $CurrentServer -State maintenance -Evacuate | Out-Null
	
# Write progress to LogFile
LogWrite "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") Put ESXi host $CurrentServer in Maintenance Mode"
	
	# Install Patch
	Get-VMHost $CurrentServer | Install-VMHostPatch -Hostpath $vibPath

# Write progress to LogFile
LogWrite "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") Installing Patch on $CurrentServer"
 
    # Reboot host
    Write-Host "Rebooting $ServerName"
    Restart-VMHost $CurrentServer -confirm:$false | Out-Null

# Write progress to LogFile
LogWrite "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") Rebooting $CurrentServer"
	
    # Wait for Server to show as down
    do {
    sleep 15
    $ServerState = (get-vmhost $ServerName).ConnectionState
    }
    while ($ServerState -ne "NotResponding")
    Write-Host "$ServerName is Down"
 
    # Wait for server to reboot
    do {
    sleep 60
    $ServerState = (get-vmhost $ServerName).ConnectionState
    Write-Host "Waiting for $ServerName to Reboot ..."
    }
    while ($ServerState -ne "Maintenance")
    Write-Host "$ServerName is back up"
 
    # Exit maintenance mode
    Write-Host "Exiting Maintenance mode"
    Set-VMhost $CurrentServer -State Connected | Out-Null
    Write-Host "** Reboot Complete **"
    Write-Host ""
	
# Write progress to LogFile
LogWrite "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") Exit ESXi host $CurrentServer from Maintenance Mode"
	
}
 
## MAIN
verifyTXTPath
foreach ($ESXiServer in $ESXiServers) {
PatchESXiServer ($ESXiServer)
}
 
# Disconnect from vCenter
Disconnect-VIServer -Server $vCenterServer -Confirm:$False

# Write progress to LogFile
LogWrite "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") Disconnect from vCenter $vCenterServer"

How to uninstall a vRO plugin

Installing vRO plugins is easy…browse…select…upload & install..restart…but uninstalling the same plugin is not so easy! There is no magic uninstall button…you need to dig a little deeper. I recently installed the wrong version of a plugin and needed to remove it before installing the correct version. Here is the procedure for the vRO appliance:

SSH to the appliance and run the following

cd /usr/lib/vco/app-server/plugins

Delete the .dar and/or .war archives that contain the plug-in that you want to remove

Restart the vRealize Orchestrator services:

service vco-server restart

service vco-configurator restart

To delete the plugin config files cd to /var/lib/vco/configuration/temp/dars/

Delete the folder.dar that contains the plug-in that you want to remove

e.g  rm -r o11nplugin-vcaccafe.dar

Finally remove the package manually.

Log in to the vRealize Orchestrator client.

In the vRealize Orchestrator client, click the Packages view.

Right-click the package to delete and click Delete element with content.

Select Delete All

that should do it!

vRA Infrastructure tab 404 error

A 404 error when accessing the infrastructure tab can be cause by a number of things. Expired certs is the most common reason. I hit this issue earlier but i’m using valid CA certs. When i logged into the VAMI interface for vRA i noticed that the IaaS service was not registered. After the customary reboots the issue persisted so i had to did a little deeper. Here is what i saw in VAMI/Services.(Not using the embedded vCO so it is intentionally disabled)

VAMI Services

To get the IaaS service registered again i had to re register IaaS with vRA using the following commands on the IaaS server

 

  • cd c:\”Program Files (x86)”\VMware\vCAC\Server\”Model Manager Data”\Cafe

Reset VMware appliance root password

Use the following procedure to change the root password for a VMware appliance. This process uses single user mode so it allow you to change the root password if you have forgotten it!

Open the VM console and reboot the appliance

 

Select the line that starts with kernel and type e to edit the line

At the end of the line, press the spacebar and type init=/bin/sh

Press Enter to exit edit mode.

On the GRUB screen, type b to boot into single-user mode.
The virtual appliance boots in single-user mode.

To change the root user password, type passwd root and follow the on-screen prompts to enter a new password for the root user

To restart the virtual appliance, type reboot and press Enter.

When the virtual appliance restarts, you can log in using the new password.

vCloud Director Error: None of the cells have a vCenter proxy service running

Came across this issue in the lab today when trying to deploy a vApp template from the vCD catalog

vCD Proxy Error

 

Did some googling and came across a post by Jason Boche here that points to an issue with the QRTZ SQL tables (Who knew!). Thanks to Jason’s post i was able to run the script below to delete some rows from the tables. Once i restarted my vCD cells i was again able to deploy vApps. Script below (modified by jason to adhere to upper case table names). Be sure to stop all vCD cells and backup the SQL DB before executing. Change the DB name to whatever your DB is called

 

USE [vcloud]
GO
delete from QRTZ_SCHEDULER_STATE
delete from QRTZ_FIRED_TRIGGERS
delete from QRTZ_PAUSED_TRIGGER_GRPS
delete from QRTZ_CALENDARS
delete from QRTZ_TRIGGER_LISTENERS
delete from QRTZ_BLOB_TRIGGERS
delete from QRTZ_CRON_TRIGGERS
delete from QRTZ_SIMPLE_TRIGGERS
delete from QRTZ_TRIGGERS
delete from QRTZ_JOB_LISTENERS
delete from QRTZ_JOB_DETAILS
go

Not sure yet what caused this but the only event i know of was a vCenter restart 2 days ago. Will update this post if i find the root cause.

vRA 6.2 Distributed IaaS deployment

I needed to deploy a distributed vRA 6.2 IaaS in the lab and didn’t find too many resources on the web so here are some of my notes! I hope to revisit this soon to do a full distributed HA deployment including NSX load balancers. For now here is a standard distributed deployment

Pre-Reqs/Assumptions

  • AD installed & available
  • vCenter/SSO installed & available
  • SQL Server installed and available and IaaS SQL pre reqs run for MSDTC
  • vRA 6.2 appliance deployed and integrated with SSO
  • AD CA configured (Will try to cover this in another post)
  • 5 Windows 2012 R2 VMs. 1 for each of the following roles – Web Server, Manager Server, DEM Orchestrator, DEM Worker, IaaS Agent
  • All windows VMs joined the AD domain
  • Run Brian Graf’s IaaS pre req script on all VMs (Pretty sure its only required on the Web & manager server but didnt take any chances!)

Here are the steps

  • Download the IaaS installer from the vRA appliance @ https://vra-FQDN:5480/i
  • First we need to install the database component. From any VM run the IaaS installer
  • Accept the ELUA and click Next
  • Enter the root credentials for the vRA appliance and select Accept Certificate and click Next
  • Choose Custom Install > IaaS Server

Installation Type

  • Select Database from the features list and enter the database instance and database name and click Next (I’m using SQL Express in the lab)

2.install DB

  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

3.DB Pre Reqs

  • Click Install on the Ready to Install Screen and Next and Finish once the Database install completes
  • Next you need to install the Website feature
  • Before installing the website component it is recommended to generate a CA signed cert for the server FQDN.
  • To do this open IIS on the Web server, select your server name and select Server Certificates
  • In the right hand pane select Create Domain Certificate

Generate Certificate

  • Enter the certificate distinguished name for the web server and click Next

Certificate Request

 

  • Click Select to choose your CA and assign a friendly name (user the web server FQDN)
  • Once complete run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Website from the feature list.
  • Choose the Default Web Site and the default port of 443
  • Under Available certificates choose the CA signed certificate you created earlier and enter the web server FQDN for the IaaS Server

web server setup

  • Because the IaaS Model Manager Data is required for the Website component to run you must also select the ModelManagerData feature
  • Enter the vCAC/vRA appliance FQDN & SSO details and also the IaaS Web Server FQDN and click Next

9. ModelManagerData

 

  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next (My firewall is disabled hence the warning. Click Bypass if you see this)

10. ModelManagerPreReqs

  • On the Server and Account Settings screen enter the password for the service account being used and a passphrase for the database and click Next and click Finish once the install completes

11. ModelManagerData Account settings

  • Next up is the Manager Service
  • On the VM you designated for the Manager Service follow the steps outlined earlier to generate a CA signed certificate
  • Once complete run the IaaS installer and again choose Custom install > IaaS Server
  • This time choose Manager Service from the feature list.
  • Enter the FQDN of the Web Server and select the certificate you created in the previous steps and click Next and click Finish

managerService Install

 

  • Next up is the DEM Orchestrator
  • On the VM you designated for the DEMO Orchestrator run the IaaS installer, choose custom > Distributed Execution Managers and click Next
  • If you have run the required pre req scripts you should get all green on the pre req checker. Click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Distributed Execution Manager Screen do the following
    • From the DEM role drop down choose Orchestrator
    • Enter a name
    • Enter a description
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Click Add and click Next and click Install and then click Finish

DEM Orchestratot Config

 

  • Repeat the above steps on the DEM Worker VM choosing the DEM Worker Role
  • Finally we will install a Proxy Agent that will be used to communicate with vRA endpoints
  • On the VM you designated for the IaaS Agent run the IaaS installer, choose custom > Proxy Agents and click Next
  • Enter the password for the service account that you are installing under and click Next
  • On the Install Proxy Agent Screen do the following
    • From the Agent Type drop down choose vSphere
    • Enter a name
    • Enter the FQDN of the Manager Server and click Test
    • Enter the FQDN of the web server and click Test
    • Enter the vCenter Endpoint FQDN
    • Click Add and click Next and click Install and then click Finish

Proxy Agent Install

  • If everything went according to plan you should now be able to log into vRA and configure tenants and resources and start using your distributed IaaS installation.

Hopefully if you’ve read this far you found this useful. I hope to post the distributed HA procedure soon once i get some free lab time!

Tagged , , ,