HashiCorp Terraform has become an industry standard, infrastructure-as-code & desired-state configuration tool for managing on-premises and cloud-based entities. If you are not familiar with Terraform, I’ve covered some early general learnings on Terraform in some posts here & here. The internal engineering team are working on a Terraform provider for VCF, so I decided to give it a spin to review its capabilities & test drive it in the lab.
First off what VCF operations is the Provider capable of supporting today:
Deploying a new VCF instance (bring-up)
Commissioning hosts
Creating network pools
Deploying a new VI Workload domain
Creating clusters
Expanding clusters
Adding users
New functionality is being added every week, and as with all new initiatives like this, customer consumption and adoption will drive innovation and progress.
The GitHub repo contains some great example files to get you started. I am going to do a few blog posts on what I’ve learned so far but for now, here are the important links you need if you would like to take a look at the provider
Before you can deploy a vSphere Lifecycle Manager (vLCM) image based cluster in VMware Cloud Foundation, you must first import an image into the Image Management Inventory in SDDC Manager. You can do this via the SDDC Manager UI for a pre existing cluster.
Or you can now use PowerVCF to import the image thanks to the addition of New-VCFPersonality (vLCM images are known as personalities in VCF hence the name of the cmdlet).
The sequence of events to be able to import an image is as follows:
Extract a vLCM image from a host that you wish to use in the workload domain. The host doesn’t need to be in the vCenter or SDDC Manager inventory
Create a temporary cluster in vCenter (must be created in a VCF workload domain) and assign the image from the previous step.
Import the image from the source cluster into SDDC Manager
To achieve step 1 we can use PowerCLI
# Variables
$sourceHostUrl = "https://sfo01-w01-esx01.sfo.rainpole.io"
$sourceHostBuild = "21495797"
$sourceHostRootPassword = "VMw@re1!"
$vcenterFQDN = "sfo-m01-vc01.sfo.rainpole.io"
$ssoUsername = "administrator@vsphere.local"
$ssoPassword = "VMw@re1!"
$vcenterDC = "sfo-m01-dc01"
$sddcManagerFQDN = "sfo-vcf01.sfo.rainpole.io"
# Retrieve the source host thumbprint
$response = [System.Net.WebRequest]::Create($sourceHostUrl)
$response.GetResponse()
$cert = $response.ServicePoint.Certificate
$sourceHostThumbprint = $cert.GetCertHashString() -replace '(..(?!$))','$1:'
# Connect to vCenter and import the image from the source host to the depot
connect-viserver -server $vcenterFQDN -user $vcenterUsername -password $vcenterPassword
$OfflineHostCredentials = Initialize-SettingsDepotsOfflineHostCredentials -HostName $sourceHostUrl -UserName "root" -Password $sourceHostRootPassword -Port 443 -SslThumbPrint $sourceHostThumbprint
$OfflineConnectionSpec = Initialize-SettingsDepotsOfflineConnectionSpec -AuthType "USERNAME_PASSWORD" -HostCredential $OfflineHostCredentials
Invoke-CreateFromHostDepotsOfflineAsync -SettingsDepotsOfflineConnectionSpec $SettingsDepotsOfflineConnectionSpec
# Create a temporary cluster and assign the image
$LcmImage = Get-LcmImage -Type BaseImage | where {$_.Version -match $sourceHostBuild}
$clusterID = (New-Cluster -Location $vcenterDC -Name 'vLCM-Cluster' -HAEnabled -DrsEnabled -BaseImage $LcmImage).ExtensionData.MoRef.Value
# Import the image to SDDDC Manager
Request-VCFToken -fqdn $sddcManagerFQDN -username $ssoUsername -password $ssoPassword
$vCenterID = (Get-VCFvCEnter | where {$_.fqdn -match $vcenterFQDN}).id
New-VCFPesonality -name "21495797" -vCenterId $vCenterID -clusterId $clusterID
That should import the new image into the SDDC Manager image repo for use creating a vLCM image based workload domain.
Since the introduction of subscription based licensing for VMware Cloud Foundation (VCF+) there are now 2 licensing modes in VCF (Perpetual or Subscription). To make it easier to identify the subscription status of the system and each workload domain we have added support for Get-VCFLicenseMode into the latest release of PowerVCF 2.3.0.1004.
Virtual NVMe isn’t a new concept. It’s been around since the 6.x days. As part of some lab work I needed to automate adding an NVMe controller and some devices to a VM. This can be accomplished using the PowerCLI cmdlets for the vSphere API.
# NVME Using PowerCLI
$vcenterFQDN = "sfo-m01-vc01.sfo.rainpole.io"
$vcenterUsername = "administrator@vsphere.local"
$vcenterPassword = "VMw@re1!"
$vms = @("sfo01-w01-esx01","sfo01-w01-esx02","sfo01-w01-esx03","sfo01-w01-esx04")
# Install the required module
Install-Module VMware.Sdk.vSphere.vCenter.Vm
# Connect to vCenter
connect-viserver -server $vcenterFQDN -user $vcenterUsername -password $vcenterPassword
# Add an NVMe controller to each VM
Foreach ($vmName in $vms)
{
$VmHardwareAdapterNvmeCreateSpec = Initialize-VmHardwareAdapterNvmeCreateSpec -Bus 0 -PciSlotNumber 0
Invoke-CreateVmHardwareAdapterNvme -vm (get-vm $vmName).ExtensionData.MoRef.Value -VmHardwareAdapterNvmeCreateSpec $VmHardwareAdapterNvmeCreateSpec
}
# Add an NVMe device
Foreach ($vmName in $vms)
{
$VmHardwareDiskVmdkCreateSpec = Initialize-VmHardwareDiskVmdkCreateSpec -Capacity 274877906944
$VmHardwareDiskCreateSpec = Initialize-VmHardwareDiskCreateSpec -Type "NVME" -NewVmdk $VmHardwareDiskVmdkCreateSpec
Invoke-CreateVmHardwareDisk -Vm (get-vm $vmName).ExtensionData.MoRef.Value -VmHardwareDiskCreateSpec $VmHardwareDiskCreateSpec
}
Once i got my head around the basics of Terraform I wanted to play with the vSphere provider to see what its was capable of. A basic use case that everyone needs is to deploy a VM. So my first use case is to deploy a VM from an OVA. The vSphere provider documentation for deploying an OVA uses William Lam’s nested ESXi OVA as an example. This is a great example of how to use the provider but seeing as I plan to play with the NSX-T provider also, I decided to use NSX-T Manager OVA as my source to deploy.
So first thing to do is setup your provider. Every provider in the Terraform registry has a Use Provider button on the provider page that pops up a How to use this provider box. This shows you what you need to put in your required_providers & provider block. In my case I will use a providers.tf file and it will look like the below example. Note you can only have one required_providers block in your configuration, but you can have multiple providers. So all required providers go in the same required_providers block and each provider has its own provider block.
To authenticate to our chosen provider (in this case vSphere) we need to provide credentials. If you read my initial post on Terraform you would have seen me mention a terraform.tfvars file which can be used for sensitive variables. We will declare these as variables later in the variables.tf file but this is where we assign the values. So my terraform.tfvars file looks like this
Next we need variables to enable us to deploy our NSX-T Manager appliance. So we create a variables.tf file and populate it with our variables. Note – variables that have a default value are considered optional and the default value will be used if no value is passed.
Now that we have our provider & variables in place we need a plan file to deploy the NSX-T Manager OVA, including the data sources we need to pull information from and the resource we are going to create.
<br />
# main.tf</p>
<p># Data source for vCenter Datacenter<br />
data "vsphere_datacenter" "datacenter" {<br />
name = var.data_center<br />
}</p>
<p># Data source for vCenter Cluster<br />
data "vsphere_compute_cluster" "cluster" {<br />
name = var.cluster<br />
datacenter_id = data.vsphere_datacenter.datacenter.id<br />
}</p>
<p># Data source for vCenter Datastore<br />
data "vsphere_datastore" "datastore" {<br />
name = var.workload_datastore<br />
datacenter_id = data.vsphere_datacenter.datacenter.id<br />
}</p>
<p># Data source for vCenter Portgroup<br />
data "vsphere_network" "mgmt" {<br />
name = var.mgmt_pg<br />
datacenter_id = data.vsphere_datacenter.datacenter.id<br />
}</p>
<p># Data source for vCenter Resource Pool. In our case we will use the root resource pool<br />
data "vsphere_resource_pool" "pool" {<br />
name = format("%s%s", data.vsphere_compute_cluster.cluster.name, "/Resources")<br />
datacenter_id = data.vsphere_datacenter.datacenter.id<br />
}</p>
<p># Data source for ESXi host to deploy to<br />
data "vsphere_host" "host" {<br />
name = var.compute_host<br />
datacenter_id = data.vsphere_datacenter.datacenter.id<br />
}</p>
<p># Data source for the OVF to read the required OVF Properties<br />
data "vsphere_ovf_vm_template" "ovfLocal" {<br />
name = var.vm_name<br />
resource_pool_id = data.vsphere_resource_pool.pool.id<br />
datastore_id = data.vsphere_datastore.datastore.id<br />
host_system_id = data.vsphere_host.host.id<br />
local_ovf_path = var.local_ovf_path<br />
ovf_network_map = {<br />
"Network 1" = data.vsphere_network.mgmt.id<br />
}<br />
}</p>
<p># Deployment of VM from Local OVA<br />
resource "vsphere_virtual_machine" "nsxt01" {<br />
name = var.vm_name<br />
datacenter_id = data.vsphere_datacenter.datacenter.id<br />
datastore_id = data.vsphere_ovf_vm_template.ovfLocal.datastore_id<br />
host_system_id = data.vsphere_ovf_vm_template.ovfLocal.host_system_id<br />
resource_pool_id = data.vsphere_ovf_vm_template.ovfLocal.resource_pool_id<br />
num_cpus = data.vsphere_ovf_vm_template.ovfLocal.num_cpus<br />
num_cores_per_socket = data.vsphere_ovf_vm_template.ovfLocal.num_cores_per_socket<br />
memory = data.vsphere_ovf_vm_template.ovfLocal.memory<br />
guest_id = data.vsphere_ovf_vm_template.ovfLocal.guest_id<br />
scsi_type = data.vsphere_ovf_vm_template.ovfLocal.scsi_type<br />
dynamic "network_interface" {<br />
for_each = data.vsphere_ovf_vm_template.ovfLocal.ovf_network_map<br />
content {<br />
network_id = network_interface.value<br />
}<br />
}</p>
<p> wait_for_guest_net_timeout = 5</p>
<p> ovf_deploy {<br />
allow_unverified_ssl_cert = true<br />
local_ovf_path = var.local_ovf_path<br />
disk_provisioning = "thin"<br />
deployment_option = var.deployment_option</p>
<p> }<br />
vapp {<br />
properties = {<br />
"nsx_role" = var.nsx_role,<br />
"nsx_ip_0" = var.nsx_ip_0,<br />
"nsx_netmask_0" = var.nsx_netmask_0,<br />
"nsx_gateway_0" = var.nsx_gateway_0,<br />
"nsx_dns1_0" = var.nsx_dns1_0,<br />
"nsx_domain_0" = var.nsx_domain_0,<br />
"nsx_ntp_0" = var.nsx_ntp_0,<br />
"nsx_isSSHEnabled" = var.nsx_isSSHEnabled,<br />
"nsx_allowSSHRootLogin" = var.nsx_allowSSHRootLogin,<br />
"nsx_passwd_0" = var.nsx_passwd_0,<br />
"nsx_cli_passwd_0" = var.nsx_cli_passwd_0,<br />
"nsx_cli_audit_passwd_0" = var.nsx_cli_audit_passwd_0,<br />
"nsx_hostname" = var.nsx_hostname<br />
}<br />
}<br />
lifecycle {<br />
ignore_changes = [<br />
#vapp # Enable this to ignore all vapp properties if the plan is re-run<br />
vapp[0].properties["nsx_role"], # Avoid unwanted changes to specific vApp properties.<br />
vapp[0].properties["nsx_passwd_0"],<br />
vapp[0].properties["nsx_cli_passwd_0"],<br />
vapp[0].properties["nsx_cli_audit_passwd_0"],<br />
host_system_id # Avoids moving the VM back to the host it was deployed to if DRS has relocated it<br />
]<br />
}<br />
}<br />
Once we have all of the above we can run the following to validate our plan
terraform plan -out=nsxt01
If your plan is successful you should see an output similar to below
Once your plan is successful run the command below to apply the plan
terraform apply nsxt01
If the stars align your NSX-T Manager appliance should deploy successfully. Once its deployed, if you were to re-run the plan you should see a message similar to below
One of the key pieces to this is the lifecycle block in the plan. The lifecycle block enables you to callout things that Terraform should ignore when it is re-applying a plan. Things like tags or other items that may get updated by other systems etc. In our case we want Terraform to ignore the vApp properties as it will try to apply password properties every time, which would entail powering down the VM, making the change, and powering the VM back on.
I was chatting with my colleague Paudie O’Riordan yesterday about PowerVCF as he was doing some testing internally and he mentioned that a great addition would be to have the ability to find, and cleanup failed tasks in SDDC Manager. Some use cases for this would be, cleaning up an environment before handing it off to a customer, or before recording a demo etc.
Currently there isnt a supported public API to delete a failed task so you have to run a curl command on SDDC Manager with the task ID. So getting a list of failed tasks and then running a command to delete each one can take time. See Martin Gustafson’s post on how to do it manually here.
I took a look at our existing code for retrieving tasks (and discovered a bug in the logic that is now fixed in PowerVCF 2.1.5!) and we have the ability to specify -status. So requesting a list of tasks with -status “failed” returns a list. So i put the script below together to retrieve a list of failed tasks, loop through them and delete them. The script requires the following inputs
SDDC Manager FQDN. This is the target that is queried for failed tasks
SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
Password for the above user
Password for the SDDC Manager appliance vcf user. This is used to run the task deletion. This is not tracked in the credentials DB so we need to pass it.
Once the above variables are populated the script does the following:
Checks for PowerVCF (minimum version 2.1.5) and installs if not present
Requests an API token from SDDC Manager
Queries SDDC Manager for the management domain vCenter Server details
Uses the management domain vCenter Server details to retrieve the SDDC Manager VM name
Queries SDDC Manager for a list of tasks in a failed state
Loops through the list of failed tasks and deletes them from SDDC Manager
Verifies the task is no longer present
Here is the script. It is also published here if you would like to enhance it
# Script to cleanup failed tasks in SDDC Manager
# Written by Brian O'Connell - Staff Solutions Architect @ VMware
#User Variables
# SDDC Manager FQDN. This is the target that is queried for failed tasks
$sddcManagerFQDN = "lax-vcf01.lax.rainpole.io"
# SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
$sddcManagerAPIUser = "administrator@vsphere.local"
$sddcManagerAPIPassword = "VMw@re1!"
# Password for the SDDC Manager appliance vcf user. This is used to run the task deletion
$sddcManagerVCFPassword = "VMw@re1!"
# DO NOT CHANGE ANYTHING BELOW THIS LINE
#########################################
# Set TLS to 1.2 to avoid certificate mismatch errors
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Install PowerVCF if not already installed
if (!(Get-InstalledModule -name PowerVCF -MinimumVersion 2.1.5 -ErrorAction SilentlyContinue)) {
Install-Module -Name PowerVCF -MinimumVersion 2.1.5 -Force
}
# Request a VCF Token using PowerVCF
Request-VCFToken -fqdn $sddcManagerFQDN -username $sddcManagerAPIUser -password $sddcManagerAPIPassword
# Disconnect all connected vCenters to ensure only the desired vCenter is available
if ($defaultviservers) {
$server = $defaultviservers.Name
foreach ($server in $defaultviservers) {
Disconnect-VIServer -Server $server -Confirm:$False
}
}
# Retrieve the Management Domain vCenter Server FQDN
$vcenterFQDN = ((Get-VCFWorkloadDomain | where-object {$_.type -eq "MANAGEMENT"}).vcenters.fqdn)
$vcenterUser = (Get-VCFCredential -resourceType "PSC").username
$vcenterPassword = (Get-VCFCredential -resourceType "PSC").password
# Retrieve SDDC Manager VM Name
if ($vcenterFQDN) {
Write-Output "Getting SDDC Manager Manager VM Name"
Connect-VIServer -server $vcenterFQDN -user $vcenterUser -password $vcenterPassword | Out-Null
$sddcmVMName = ((Get-VM * | Where-Object {$_.Guest.Hostname -eq $sddcManagerFQDN}).Name)
}
# Retrieve a list of failed tasks
$failedTaskIDs = @()
$ids = (Get-VCFTask -status "Failed").id
Foreach ($id in $ids) {
$failedTaskIDs += ,$id
}
# Cleanup the failed tasks
Foreach ($taskID in $failedTaskIDs) {
$scriptCommand = "curl -X DELETE 127.0.0.1/tasks/registrations/$taskID"
Write-Output "Deleting Failed Task ID $taskID"
$output = Invoke-VMScript -ScriptText $scriptCommand -vm $sddcmVMName -GuestUser "vcf" -GuestPassword $sddcManagerVCFPassword
# Verify the task was deleted
Try {
$verifyTaskDeleted = (Get-VCFTask -id $taskID)
if ($verifyTaskDeleted -eq "Task ID Not Found") {
Write-Output "Task ID $taskID Deleted Successfully"
}
}
catch {
Write-Error "Something went wrong. Please check your SDDC Manager state"
}
}
Disconnect-VIServer -server $vcenterFQDN -Confirm:$False
In Part 1 of this series we saw how to retrieve a sessionId from the Site Recovery Manager VAMI interface using Postman & Powershell. In this post we will use that sessionId to replace the appliance SSL certificate using the API. To start we again use the VAMI UI to inspect the endpoint URL being used for certificate replacement by doing a manual replacement. In this case the URL is:
Site Recovery Manager expects the certificate in P12 format so I used CertGen to create the cert format needed. When using the UI you browse to the cert file and it uploads in the browser along with the certificate passphrase. Behind the scenes it is then base64 encoded, so you need to do this before using the API.
I’ve recently been doing a lot of work with VMware Site Recovery Manager (SRM) and vSphere Replication (vSR) with VMware Cloud Foundation. Earlier this year we (Ken Gould & I) published an early access design for Site Protection & Recovery for VMware Cloud Foundation 4.2. We have been working to refresh & enhance this design for a new release. Part of this effort includes trying to add some automation to assist with the manual steps to speed up time to deploy. SRM & vSR do not have publicly documented VAMI APIs so we set about trying to automate the configuration with a little bit of reverse engineering.
As with most APIs, whether public or private, you must authenticate before you can run an API workflow, so the first task is figuring out how the authentication to perform a workflow works. Typically, if you hit F12 in your browser you will get a developer console that exposes what goes on behind the scenes in a browser session. So to inspect the process, use the browser to perform a manual login, and review the header & response tabs in the developer view. This exposes the Request URL to use, the method (POST) and the required headers (accept: application/json)
The Response tab shows a sessionId which can be used for further configuration API calls in the headers as dr.config.service.sessionid
So with the above information you can use an API client like Postman to retrieve a sessionId with the URL & headers like this
And your VAMI admin user and password in JSON format in the body payload
You can also use the information to retrieve a sessionId using PowerShell
Within a VMware Cloud Foundation instance, SDDC Manager is used to manage the lifecycle of passwords (or credentials). While we provide the ability to rotate (either scheduled or manually) currently there is no easy way to check when a particular password is due to expire, which can lead to appliance root passwords expiring, which will cause all sorts of issues. The ability to monitor expiry is something that is being worked on, but as a stop gap I put together the script below which leverages PowerVCF and also a currently undocumented API for validating credentials.
The script has a function called Get-VCFPasswordExpiry that accepts the following parameters
-fqdn (FQDN of the SDDC Manager)
-username (SDDC Manager Username – Must have the ADMIN role)
-password (SDDC Manager password)
-resourceType (Optional parameter to specify a resourceType. If not passed, all resources will be checked. If passed (e.g. VCENTER) then only that resourceType will be checked. Supported resource types are
PowerVCF is a requirement. If you dont already have it run the following
Install-Module -Name PowerVCF
The code takes a while to run as it needs to do the following to check password expiry
Connect to SDDC Manager to retrieve an API token
Retrieve a list of all credentials
Using the resourceID of each credential
Perform a credential validation
Wait for the validation to complete
Parse the results for the expiry details
Add all the results to an array and present in a table (Kudos to Ken Gould for assistance with the presentation of this piece!)
In this example script I am returning all non SERVICE user accounts regardless of expiry (SERVICE account passwords are system managed). You could get more granular by adding something like this to only display accounts with passwords due to expire in less than 14 days
if ($validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry -lt 14) {
Write-Output "Password for username $($validationTaskResponse.validationChecks.username) expires in $($validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry) days"
}
LifeOfBrianOC 