Cleanup Failed Tasks in SDDC Manager

I was chatting with my colleague Paudie O’Riordan yesterday about PowerVCF as he was doing some testing internally and he mentioned that a great addition would be to have the ability to find, and cleanup failed tasks in SDDC Manager. Some use cases for this would be, cleaning up an environment before handing it off to a customer, or before recording a demo etc.

Currently there isnt a supported public API to delete a failed task so you have to run a curl command on SDDC Manager with the task ID. So getting a list of failed tasks and then running a command to delete each one can take time. See Martin Gustafson’s post on how to do it manually here.

I took a look at our existing code for retrieving tasks (and discovered a bug in the logic that is now fixed in PowerVCF 2.1.5!) and we have the ability to specify -status. So requesting a list of tasks with -status “failed” returns a list. So i put the script below together to retrieve a list of failed tasks, loop through them and delete them. The script requires the following inputs

  • SDDC Manager FQDN. This is the target that is queried for failed tasks
  • SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
  • Password for the above user
  • Password for the SDDC Manager appliance vcf user. This is used to run the task deletion. This is not tracked in the credentials DB so we need to pass it.

Once the above variables are populated the script does the following:

  • Checks for PowerVCF (minimum version 2.1.5) and installs if not present
  • Requests an API token from SDDC Manager
  • Queries SDDC Manager for the management domain vCenter Server details
  • Uses the management domain vCenter Server details to retrieve the SDDC Manager VM name
  • Queries SDDC Manager for a list of tasks in a failed state
  • Loops through the list of failed tasks and deletes them from SDDC Manager
  • Verifies the task is no longer present

Here is the script. It is also published here if you would like to enhance it

# Script to cleanup failed tasks in SDDC Manager
# Written by Brian O'Connell - Staff Solutions Architect @ VMware

#User Variables
# SDDC Manager FQDN. This is the target that is queried for failed tasks
$sddcManagerFQDN = "lax-vcf01.lax.rainpole.io"
# SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
$sddcManagerAPIUser = "administrator@vsphere.local"
$sddcManagerAPIPassword = "VMw@re1!"
# Password for the SDDC Manager appliance vcf user. This is used to run the task deletion
$sddcManagerVCFPassword = "VMw@re1!"



# DO NOT CHANGE ANYTHING BELOW THIS LINE
#########################################

# Set TLS to 1.2 to avoid certificate mismatch errors
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# Install PowerVCF if not already installed
if (!(Get-InstalledModule -name PowerVCF -MinimumVersion 2.1.5 -ErrorAction SilentlyContinue)) {
    Install-Module -Name PowerVCF -MinimumVersion 2.1.5 -Force
}

# Request a VCF Token using PowerVCF
Request-VCFToken -fqdn $sddcManagerFQDN -username $sddcManagerAPIUser -password $sddcManagerAPIPassword

# Disconnect all connected vCenters to ensure only the desired vCenter is available
if ($defaultviservers) {
    $server = $defaultviservers.Name
    foreach ($server in $defaultviservers) {            
        Disconnect-VIServer -Server $server -Confirm:$False
    }
}

# Retrieve the Management Domain vCenter Server FQDN
$vcenterFQDN = ((Get-VCFWorkloadDomain | where-object {$_.type -eq "MANAGEMENT"}).vcenters.fqdn)
$vcenterUser = (Get-VCFCredential -resourceType "PSC").username
$vcenterPassword = (Get-VCFCredential -resourceType "PSC").password

# Retrieve SDDC Manager VM Name
if ($vcenterFQDN) {
    Write-Output "Getting SDDC Manager Manager VM Name"
    Connect-VIServer -server $vcenterFQDN -user $vcenterUser -password $vcenterPassword | Out-Null
    $sddcmVMName = ((Get-VM * | Where-Object {$_.Guest.Hostname -eq $sddcManagerFQDN}).Name)              
}

# Retrieve a list of failed tasks
$failedTaskIDs = @()
$ids = (Get-VCFTask -status "Failed").id
Foreach ($id in $ids) {
    $failedTaskIDs += ,$id
}
# Cleanup the failed tasks
Foreach ($taskID in $failedTaskIDs) {
    $scriptCommand = "curl -X DELETE 127.0.0.1/tasks/registrations/$taskID"
    Write-Output "Deleting Failed Task ID $taskID"
    $output = Invoke-VMScript -ScriptText $scriptCommand -vm $sddcmVMName -GuestUser "vcf" -GuestPassword $sddcManagerVCFPassword

# Verify the task was deleted    
    Try {
    $verifyTaskDeleted = (Get-VCFTask -id $taskID)
    if ($verifyTaskDeleted -eq "Task ID Not Found") {
        Write-Output "Task ID $taskID Deleted Successfully"
    }
}
    catch {
        Write-Error "Something went wrong. Please check your SDDC Manager state"
    }
}
Disconnect-VIServer -server $vcenterFQDN -Confirm:$False

As always, comments/feedback welcome!

Site Protection & Disaster Recovery for VMware Cloud Foundation Validated Solution

Along with the release of VMware Cloud Foundation 4.3.1, we are excited to announce the general availability of the Site Protection & Disaster Recovery for VMware Cloud Foundation Validated Solution. The solution documentation, intro and other associated collateral can be found on the Cloud Platform Tech Zone here.

The move from VMware Validated Designs to VMware Validated Solutions has been covered by my team mate Gary Blake in detail here so I wont go into that detail here. Instead I will concentrate on the work Ken Gould and I (along with a supporting team) have been working to deliver for the past few months.

The Site Protection & Disaster Recovery for VMware Cloud Foundation Validated Solution includes the following to deliver an end-to-end validated way to protect your mission critical applications. You get a set of documentation that is tailored to the solution that includes: design objectives, a detailed design including not just design decisions, but the justifications & implications of those decisions, detailed implementation steps with PowerShell alternatives for some steps to speed up time to deploy, operational guidance on how to use the solution once its deployed, solution interoperability between it and other Validated Solutions, an appendix containing all the solution design decisions in one easy place for review, and finally, a set of frequently asked questions that will be updated for each release.

Disaster recovery is a huge topic for everyone lately. Everything from power outages to natural disasters to ransomware and beyond can be classed as a disaster, and regardless of the type of disaster you must be prepared. To adequately plan for business continuity in the event of a disaster you must protect your mission critical applications so that they may be recovered. In a VMware Cloud Foundation environment, cloud operations and automation services are delivered by vRealize Lifecycle Manager, vRealize Operations Manager & vRealize Automation, with authentication services delivered by Workspace ONE Access.

To provide DR for our mission critical apps we leverage 2 VCF instances with NSX-T federation between them. The primary VCF instance runs the active NSX-T global manager and the recovery VCF instance runs the standby NSX-T global manager. All load balancing services are served from the protected instance, with a standby load balancer (disconnected from the recovery site NSX Tier-1 until required, to avoid IP conflicts) in the recovery instance. Using our included PowerShell cmdlets you can quickly create and configure the standby load balancer to mimic your active load balancer, saving you a ton of manual UI clicks.

In the (hopefully never) event of the need to failover the cloud management applications, you can easily bring the standby load balancer online to enable networking services for the failed over applications.

Using Site recovery Manager (SRM) you can run planned migrations or disaster recovery migrations. With a single set of SRM recovery plans, regardless of the scenario, you will be guided through the recovery process. In this post I will cover what happens in the event of a disaster.

When a disaster occurs on the protected site (once the panic subsides) there are a series of tasks you need to perform to bring those mission critical apps back online.

First? Fix the network! Log into the passive NSX Global Manager (GM) on the recovery site and promote the GM to Active. (Note: This can take about 10-15 mins)

To cover the case of an accidental “Force Active” click..we’ve built in the “Are you absolutely sure this is what you want to do?” prompt!

Once the promotion operation completes our standby NSX GM is now active, and can be used to manage the surviving site NSX Local Manager (LM)

Once the recovery site GM is active we need to ensure that the cross-instance NSX Tier-1 is now directing the egress traffic via the recovery site. To do this we must update the locations on the Tier-1. Navigate to GM> Tier-1 gateways > Cross Instance Tier-1. Under Locations, make the recovery location Primary.

The next step is to ensure we have an active load balancer running in the recovery site to ensure our protected applications come up correctly. To do this log into what is now our active GM, select the recovery site NSX Local Manager (LM), and navigate to Networking > Load Balancing. Edit the load balancer and attach it to the recovery site standalone Tier-1.

At this point we are ready to run our SRM recovery plans. The recommended order for running the recovery plans (assuming you have all of the protected components listed below) is as follows. This ensures lifecycle & authentication services (vRSLCM & WSA) are up before the applications that depend on them (vROPS & vRA)

  • vRSLCM – WSA – RP
  • Intelligent Operations Management RP
  • Private Cloud Automation RP

I’m not going to go through each recovery plan in detail here. They are documented in the Site Protection and Disaster Recovery Validated Solution. In some you will be prompted to verify this or that along the way to ensure successful failover.

The main thing in a DR situation is, DO NOT PANIC. And what is the best way to getting to a place where you DO NOT PANIC? Test your DR plans…so when you see this…

Your reaction is this…

BookReview: Rest — Fresh Perception

Trust the plan…test the plan…relax…you have a plan!

Hopefully this post was useful..if you want to learn more please reach out in the comments…if you’re attending VMworld and would like to learn more or ask some questions, please drop into our Meet The Experts session on Thursday.

Take a look at Ken’s post on the Planning & Preparation Workbook for this validated solution for more details.

PowerVCF 2.1.1 Now Available with Support for VMware Cloud Foundation 4.1

PowerVCF 2.1.1 is now available on the PowerShell Gallery here or from GitHub here. This release includes support for VMware Cloud Foundation 4.1 with some new and updated cmdlets. Highlights below

 

Category cmdlet Name Description Comment
Users and Groups Get-VCFCredential Retrieves a list of credentials. UPDATE: Added support for the vRealize Suite credentials
Bundles Start-VCFBundleUpload Starts upload of bundle to SDDC Manager UPDATE: Allows the import of a bundle based on offline download.
Federation New-VCFFederationInvite Invite new member to VCF Federation UPDATE: Added support to specify if the new system is a MEMBER or CONTROLLER.
SDDC Start-CloudBuilderSDDCValidation Starts validation on VMware Cloud Builder UPDATE: Added support for individual validation tasks.
Workspace ONE Access Get-VCFWSA Get details of the existing Workspace ONE Access NEW
vRealize Automation Get-VCFvRA Get details of the the existing vRealize Automation NEW
vRealize Operations Get-VCFvROPs Get details of the existing vRealize Operations Manager NEW
vRealize Operations Set-VCFvROPs Connect or disconnect Workload Domains to vRealize Operations Manager NEW
vRealize Log Insight Get-VCFvRLI Get details of the existing vRealize Log Insight NEW
vRealize Log Insight Set-VCFvRLI Connect or disconnect Workload Domains to vRealize Log Insight NEW
vRealize Suite Lifecycle Get-VCFvRSLCM Get details of the existing vRealize Suite Lifecycle Manager UPDATE: Fixed an issue with the API URI and addressed response output

VMware Cloud Foundation Bringup With Signed Certs on ESXi Hosts

Traditionally VMware Cloud Foundation (VCF) has followed the hybrid approach when it comes to SSL certificate management. Hybrid mode essentially means using CA signed certs for the vCenter Server machineSSL cert, and VMCA signed certs for the solution user certs. In this mode, ESXi host certs are VMCA managed also. You then have the option to integrate with an external Microsoft CA or continue to use VMCA for all certs. If you decide to integrate with a Microsoft CA, ESXi host certs remain VMCA managed. This is not always ideal as some customers require all components on the network to be signed by a known & trusted CA. Up until the recent 4.1 VMware Cloud Foundation (VCF) release it was not possible to use custom CA signed certs on your ESXi hosts, as hybrid mode would overwrite your CA signed ESXi certs with VMCA signed certs. There is a great blog post here on how to manually enable CA signed certs here but with VCF 4.1 it is now supported to do this via the API during bringup. The procedure is as follows:

 

  1. Install the ESXi hosts that will be used for bringup with the ESXi version on the Bill Of Materials for 4.1
  2. Install your custom CA signed certs on each host that will be used for the management domain
    1. Log in to the ESXi Shell, either directly from the DCUI or from an SSH client, as a user with administrator privileges.
    2. In the directory /etc/vmware/ssl, rename the existing certificates using the following commands.
      mv rui.crt orig.rui.crt 
      mv rui.key orig.rui.key
      
    3. Copy the certificates that you want to use to /etc/vmware/ssl.
    4. Rename the new certificate and key to rui.crt and rui.key.
    5. Restart the host management agents by running the following commands
 /etc/init.d/hostd restart /etc/init.d/vpxa restart 

Repeat the above steps for all management domain hosts

To ensure that SDDC Manager is aware that you are using custom certs you need to add a flag in the bringup json along with the PEM encoded signing chain certificate, so that it is added to the SDDC Manager keystore. This will ensure the certificates are trusted. The API guide for 4.1 provides an example json spec here. Pay particular attention to this section

securitySpec" : {
"esxiCertsMode" : "One among:Custom, VMCA",
"rootCaCerts" : [ {
"alias" : "string",
"certChain" : [ "string" ]
} ]
}

So to enable support for signed certs you would set this section as follows (Substituting your signing CA chain)

 securitySpec" : { 
"esxiCertsMode" : "Custom", 
"rootCaCerts" : [ { 
"alias" : "Rainpole-CA", 
"certChain" : [ "-----BEGIN CERTIFICATE-----
MIIDczCCAlugAwIBAgIQI9xwbTkI9J5GhMffcP5CHDANBgkqhkiG9w0BAQsFADBM
MRIwEAYKCZImiZPyLGQBGRYCaW8xGDAWBgoJkiaJk/IsZAEZFghyYWlucG9sZTEc
MBoGA1UEAxMTcmFpbnBvbGUtZGMwMXJwbC1DQTAeFw0yMDAzMzAxNDQ2MTNaFw0y
NTAzMzAxNDU2MTNaMEwxEjAQBgoJkiaJk/IsZAEZFgJpbzEYMBYGCgmSJomT8ixk
ARkWCHJhaW5wb2xlMRwwGgYDVQQDExNyYWlucG9sZS1kYzAxcnBsLUNBMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzpwkz7aPlQcfevcCelHc9DPswHkd
kjY96Vh3GvYlesaVEcy/q/BOvvh3KgLMLy8r7cy2cNPO3FANKOfqVdVx3ghfEUyL
g61W9BskAlwryzJRmjhOJJVqvB8CWjy+eCp7MejHGdEud6WdEvK8CaBcPngEg0KM
eLRNLGe8OCw8yY4GTrjU+H7PYQZtyD0kxxy5f48ueaDXat4ENRGcAuHEfCoMGfaR
bDue1OO4diHd900bCym5ggBNX0jhRudNULXPTayZl2ksImV0+QkaVeptQImXfCgb
kgnHQJ5CxK26up7fB5eAsmGLAsJLBnHuM7P9xvV09EvWjFCgLX/oBBDYTQIDAQAB
o1EwTzALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU7oOq
QBK8yg8mHnAfb+u6/GO0ZUcwEAYJKwYBBAGCNxUBBAMCAQAwDQYJKoZIhvcNAQEL
BQADggEBALYxZGj4vWjFDN1atOUsBx2jrmxbExgMAyRpNlSc2aj+7vzxHxUW5VbX
x9nc/BfkTiCK6c7Y9VYb+mgjb8z0kNv58sT4ar1yIl1n63VOCoyyLcaFB8HyEJpD
wUhz4RNPoSijZMpm+M5EuSLfWlhEJo7N8sLqHgvvk1dFpbK8fIHbPS5KJwJibbPe
w9UuNRdcxN9hFWKBC0SvfgX+1CJxVdvgfi65rSHPuWinJzrXXdH999DfpDESRzwH
0pqE3GtMCt1Nqalp2QJFdahbT+kxj7QWHTjUylSENDHjdln7a8WH8RGxvEy/97YZ
+crXmxvQ/bAgHk9vcRERbRjfyIs7v88=
-----END CERTIFICATE-----" ] } ] } 

 

 

You can then follow the steps outlined in the API guide to deploy the management domain using the Cloud Builder API. Note that once custom mode is enabled, all future workload domains that you create must also use signed certs.

Where Are My VMware Cloud Foundation Logs?

From time to time we all need to look at logs, whether its a failed operation or to trace who did what when. In VMware Cloud Foundation there are many different logs, each one serving a different purpose. Its not always clear which log you should look at for each operation so here is a useful reference table.

Log TypeVM Locationlog Location
BringUpCloud Builder

JSON Generator – /opt/vmware/sddc-support/cloud_admin_tools/logs/JsonGenerator.log

Platform Audit – /opt/vmware/sddc-support/cloud_admin_tools/logs/PlatformAudit.log

Bringup – /var/log/vmware/vcf/bringup/vcf-bringup-debug.log
LicensingSDDC Manager/var/log/vmware/vcf/operationsmanager/operationsmanager.log
Network PoolSDDC Manager/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
Host Commission/DecommissionSDDC Manager/var/log/vmware/vcf/operationsmanager/operationsmanager.log
VI (WLD domain)SDDC Manager/var/log/vmware/vcf/domainmanager/domainmanager.log
vRLISDDC Manager/var/log/vmware/vcf/domainmanager/domainmanager.log
/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
vROPSSDDC Manager/var/log/vmware/vcf/domainmanager/domainmanager.log
/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
vRASDDC Manager/var/log/vmware/vcf/domainmanager/domainmanager.log
/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
vRSLCMSDDC Manager/var/log/vmware/vcf/domainmanager/domainmanager.log
/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
Upgrade: /var/log/vmware/vcf/lcm/lcm.log
 
vRSLCM/var/log/vrlcm/vmware_vrlcm.log
LCMSDDC Manager/var/log/vmware/vcf/lcm/lcm.log
API LoginSDDC Manager/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
SoSSDDC Manager/var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log
Certificate OperationsSDDC Manager/var/log/vmware/vcf/operationsmanager/operationsmanager.log

PowerVCF 2.0 Authentication Changes

One of the many major enhancements in VMware Cloud Foundation 4.0 is a switch from basic authentication to token based authentication for the VCF API.

Basic authentication is a header field in the form of Authorization: Basic <credentials>, where credentials is the base64 encoding of a username and password. The credentials are not encrypted, therefore Basic Authentication is not the industry standard for API authentication.

VCF 4.0 has moved to using token based authentication (JWT Tokens to be exact) for securing the API. The token implementation is as follows:

  1. An authorized user executes a POST API call to /v1/tokens
  2. The response contains an access token and a refresh token
    1. The access token is valid for 1 hour
      1. The access token is passed in every API call header in the form of Authorization: Bearer <access token>
    2. The refresh token is valid for 24 hours
      1. The refresh token is used to request a new access token once it has expired

PowerVCF 2.0 abstracts all of this in the following way:

  • An authorized user connects to SDDC Manager to request the tokens by running:

Connect-VCFManager -fqdn sfo-vcf01.sfo.rainpole.io -username svc-vcf-api@rainpole.io -password VMw@re1!

  • The access & refresh tokens are stored in memory and used when running subsequent API calls. As each API call is executed PowerVCF checks the expiry of the access token. If the access token is about to expire, it uses the refresh token to request a new access token and proceeds with the API call. So the user does not need to worry about token management.

We have also introduced roles that can be assigned to users. Initially we have ADMIN & OPERATOR, with more roles planned for a future release.

ADMIN = Full Administrator Access to all APIs

OPERATOR = All Access except Password Management, User Management, Backup Management

To request an API token you must have a user account that is assigned either the ADMIN or OPERATOR role in SDDC Manager. The default administrator@vsphere.local user is assigned the ADMIN role during bringup but it is advisable to add additional users for performing day to day tasks.

Once you have a user added you can then authenticate with SDDC Manager to retrieve your access & refresh tokens.

Tip: You can connect using the administrator@vsphere.local user to add new users using PowerVCF. You can use the New-VCFUser PowerVCF cmdlet to create the user and assign a role like so:


Connect-VCFManager -fqdn sfo-vcf01.sfo.rainpole.io -username administrator@vsphere.LOCAL -password VMw@re1!

New-VCFUser -user vcf-admin@rainpole.io -role ADMIN

Once your user is configured PowerVCF will do the rest when it comes to managing the API access tokens.

 

Announcing PowerVCF 2.0

I’m happy to announce the availability of PowerVCF 2.0. This version of PowerVCF is compatible with VMware Cloud Foundation 4.0 and above. Due to some API security enhancements in VCF 4.0 around the use of API tokens for authentication the module has been refactored to leverage access & refresh tokens (more on that here). For that reason if you would like to use PowerVCF for VCF 3.9.x you should continue to use PowerVCF 1.2 .

PowerVCF 2.0 is published to the PowerShell Gallery here https://www.powershellgallery.com/packages/PowerVCF/2.0.0

Whats new in PowerVCF 2.0

Along with a number of new or modified cmdlets the following enhancements have been made:

  • Grouped cmdlets based on order of API documentation
  • Code hygiene

The following table provides a detailed breakdown of all the changes for this release. Thanks to my colleague @GaryJBlake for doing most of the work on this release and for pulling this list together!

Category cmdlet Name Description Comment
Backup and Restore Start-VCFRestore Starts the restore process of SDDC Manager NEW
Backup and Restore Get-VCFRestoreTasks Gets the status of the restore process NEW
Connectivity Connect-VCFManager Create authentication header for SDDC Manager appliance UPDATED – Support the new token / bearer authentication model and basicAuth switch for restore process
Connectivity Connect-CloudBuilder Create authentication header for Cloud Builder appliance NEW
Certificates Get-VCFCertificateAuthority Get Certificate Authority information UPDATED – Added support for getting the details by id
Certificates Remove-VCFCertificateAuthority Deletes Certificate Authority configuration NEW
Certificates Get-VCFCertificate View certificate of all the resources in a domain UPDATED – Added support for get certificate details by resource
Credentials Get-VCFCredential Get the credentials UPDATED- Added support for getting the details by id
Credentials Stop-VCFCredentialTask Cancels a failed update or rotate passwords task RENAMED – From Cancel-VCFCredentialTask
Credentials Restart-VCFCredentialTask Retry a failed rotate/update passwords task RENAMED – From Retry-VCFCredentialTask
Hosts Commission-VCFHost Commissions a list of hosts UPDATED – Added support for validating the input spec for host operations (-validate switch)
NSX-T Edge Clusters Get-VCFEdgeCluster Get an Edge Cluster NEW
NSX-T Edge Clusters New-VCFEdgeCluster creates an NSX-T edge cluster NEW
Personalities Get-VCFPersonality Get the vSphere Lifecycle Manager Personalities NEW
SDDC (Cloud Builder) Get-CloudBuilderSDDC Retrieve all SDDCs NEW
SDDC (Cloud Builder) Start-CloudBuilderSDDC Create SDDC NEW
SDDC (Cloud Builder) Restart-CloudBuilderSDDC Retry failed SDDC creation NEW
SDDC (Cloud Builder) Get-CloudBuilderSDDCValidation Get all SDDC specification validations NEW
SDDC (Cloud Builder) Start-CloudBuilderSDDCValidation Validate SDDC specification before creation NEW
SDDC (Cloud Builder) Stop-CloudBuilderSDDCValidation Cancel SDDC specification validation NEW
SDDC (Cloud Builder) Restart-CloudBuilderSDDCValidation Retry SDDC validation NEW
System Prechecks Start-VCFSystemPrecheck Perform System Precheck RENAMED – From Start-PreCheckVCFSystem
System Prechecks Get-VCFSystemPrecheckTask Get System Precheck Task RENAMED – From Get-PreCheckVCFSystemTask
Tasks Restart-VCFTask Retry a previously failed task RENAMED – From Retry-VCFTask
Users Get-VCFRole Get all roles NEW
Users Get-VCFUser Get all Users NEW
Users New-VCFUser Adds a new user NEW
Users New-VCFServiceUser Adds a new service user NEW
Users Delete-User Deletes a user NEW
vRealize Suite Lifecycle Manager Reset-VCFvRSLCM Redeploy vRealize Suite Lifecycle Manager NEW
vRealize Suite Lifecycle Manager New-VCFvRSLCM Validate the input specification for vRealize Suite Lifecycle Manager deployment UPDATED – Added support for validating the json spec (-validate switch).

PowerShell Script to Configure an NSX-T Load Balancer for the vRealize Suite & Workspace ONE Access

As part of my role in the VMware Hyper-converged Business Unit (HCIBU) I spend a lot of time working with new product versions testing integrations for next-gen VMware Validated Designs and Cloud Foundation. A lot of my focus is on Cloud Operations and Automation (vROPs, vRLI, vRA etc) and consequently I regularly need to deploy environments to perform integration testing. I will typically leverage existing automation where possible and tend to create my own when i find gaps. Once such gap was the ability to use PowerShell to interact with the NSX-T API. For anyone who is familiar with setting up a load balancer for the vRealize Suite in NSX-T – there are a lot of manual clicks required. So i set about creating some PowerShell functions to make it a little less tedious and to speed up getting my environments setup so i could get to the testing faster.

There is comprehensive NSX-T API documentation posted on code.vmware .com that I used to decipher the various API endpoints required to complete the various tasks:

  • Create the Load Balancer
  • Create the Service Monitors
  • Create the Application Profiles
  • Create the Server Pools
  • Create the Virtual Servers

The result is a PowerShell module with a function for each of the above and a corresponding JSON file that is read in for the settings for each function. I have included a sample JSON file to get you started. Just substitute your values.

Note: You must have a Tier-1 & associated segments created. (I’ll add that functionality when i get a chance!)

PowerShell Module, Sample JSON & Script are posted to Github here

Create a multi pNIC VMware Cloud Foundation NSX-V Workload Domain with PowerVCF

Hopefully by now you’ve seen my earlier posts about the new PowerShell module for the VMware Cloud Foundation API. If not i’d suggest reviewing these before reading on

With the release of VMware Cloud Foundation 3.9.1 it is now supported, via the API only, to use more than 2 physical NICs (pNICs) per host. In fact the API now supports up to three vSphere Distributed switches and six physical NICs, providing more flexibility to support high performance use cases and physical traffic separation.

There is a tech note that goes into more detail on the use cases for more than 2 pNICs and it also shows how this works using PostMan but we can also achieve this using PowerVCF.

The workflow using PowerVCF is the same as my earlier example for creating a workload domain. The only difference is the content in the JSON file.

Note: There is a validation API to validate the JSON you are passing before making the submission. PowerVCF dynamically formats the validation JSON as the formatting is slightly different to what you submit to create the workload domain.

To get you started there is a sample JSON file with the required formatting. Here is a snapshot of what it looks like

{
  "domainName": "PowerVCF",  
  "vcenterSpec": {  
    "name": "sfo01w01vc01",  
    "networkDetailsSpec": {  
       "ipAddress": "172.16.225.64",  
       "dnsName": "sfo01w01vc01.sfo01.rainpole.local",  
       "gateway": "172.16.225.1",  
       "subnetMask": "255.255.255.0"
     },  
     "rootPassword": "VMw@re1!",  
     "datacenterName": "PowerVCF-DC"  
   },  
   "computeSpec": {  
      "clusterSpecs": [ {  
          "name": "Cluster1",  
          "hostSpecs": [ {  
              "id": "d0693b58-4012-4387-92ed-721cfa709e44",
              "license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ]  
               }  
            }, {  
              "id": "7006bec4-fccb-49a0-bff6-fd56c807d26a",
              "license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ]  
               }  
            }, {  
              "id": "cc257a80-e179-4297-bf7e-179a0944bbab",
              "license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ] 
               } 
           } ],     
    "datastoreSpec": {  
        "vsanDatastoreSpec": {  
            "failuresToTolerate": 1,  
            "licenseKey": "BBBBB-BBBBB-BBBBB-BBBBB-BBBBB",
            "datastoreName": "vSanDatastore" 
         }  
     },  
     "networkSpec": { 
         "vdsSpecs": [ { 
             "name": "SDDC-Dswitch-Private1", 
             "portGroupSpecs": [ {  
                 "name": "SDDC-DPortGroup-Mgmt", 
                 "transportType": "MANAGEMENT" 
             }, { 
                 "name": "SDDC-DPortGroup-VSAN",  
                 "transportType": "VSAN" 
             }, {  
                 "name": "SDDC-DPortGroup-vMotion", 
                 "transportType": "VMOTION" 
             } ] 
          },  
          {  
             "name": "SDDC-Dswitch-Private2", 
             "portGroupSpecs": [ { 
                "name": "SDDC-DPortGroup-Public", 
                "transportType": "PUBLIC"  } ] 
           } 
        ],  
        "nsxClusterSpec": { 
           "nsxVClusterSpec": {  
              "vlanId": 2237,  
              "vdsNameForVxlanConfig": "SDDC-Dswitch-Private1"  
            }  
          }  
        }  
      } ] 
   }, 
  "nsxVSpec" : {
    "nsxManagerSpec" : {
      "name" : "sfo01w01nsx01",
      "networkDetailsSpec" : {
        "ipAddress" : "172.16.225.66",
        "dnsName" : "sfo01w01nsx01.sfo01.rainpole.local",
        "gateway" : "172.16.225.1",
        "subnetMask" : "255.255.255.0"
      }
    },
    "nsxVControllerSpec" : {
      "nsxControllerIps" : [ "172.16.225.121", "172.16.225.122", "172.16.225.123" ],
      "nsxControllerPassword" : "VMw@re123456!",
      "nsxControllerGateway" : "172.16.225.1",
      "nsxControllerSubnetMask" : "255.255.255.0"
    },
    "licenseKey" : "CCCCC-CCCCC-CCCCC-CCCCC-CCCCC",
    "nsxManagerAdminPassword" : "VMw@re1!",
    "nsxManagerEnablePassword" : "VMw@re1!"
  }
}

You can see that the magic happens in the hostNetworkSpec section where you map each vmnic to a vdsName

<p>"hostNetworkSpec": { "vmNics": [ { "id": "vmnic0", "vdsName": "SDDC-Dswitch-Private1" }, { "id": "vmnic1", "vdsName": "SDDC-Dswitch-Private1" }, { "id": "vmnic2", "vdsName": "SDDC-Dswitch-Private2" }, { "id": "vmnic3", "vdsName": "SDDC-Dswitch-Private2" } ] }</p>

So please try it out and let us know how it goes!

Create a new VMware Cloud Foundation Workload Domain with PowerVCF

So now that we have a PowerShell module for the VMware Cloud Foundation API, just what can we do with it? Well in this example we will create an NSX-V backed VMware Cloud Foundation workload domain, all using PowerVCF to interact with the API. Now all of this could obviously be wrapped up in a single script but I’m going to show you each step, with some tips along the way.

I will be making the assumption that you are familiar with VMware Cloud Foundation Concepts. If not please review the documentation here.

So once you have the initial VCF bringup completed you need to add a workload domain(s) to service our workloads. In my example below I have a management domain only.

And i have only the 4 hosts that are part of the management domain in my inventory. So i need to add new hosts to my inventory before i can create a new workload domain.

The sequence of events is as follows:

  • Install the PowerVCF Module
  • Connect to SDDC Manager
  • Create a network pool
  • Commission hosts
  • Create Workload domain

Install the PowerVCF Module from the PowerShell Gallery

  • Open PowerShell
  • Run the following to install the module

Install-Module -Name PowerVCF

Connect to SDDC Manager

  • To establish a session with SDDC Manager run the following

Connect-VCFManager -fqdn sddc-manager.sfo01.rainpole.local -username admin -password VMw@re1!

Create a network pool

The first thing you need before you can commission new hosts is to create a new network pool, which will include the vSAN & vMotion network details for this workload domain cluster.

To create a new network pool do the following:

  • Before you can create a network pool you first need to create the json body that will be passed in.

TIP: The PowerVCF Module includes a folder of sample json files to get you started

Here is the json format required for creating a vSAN network pool (Please use the same json with the module rather than copying from here as formatting is probably messed up!)


{
"name": "sfo01w01-cl01",
"networks": [
{
"type": "VSAN",
"vlanId": 2240,
"mtu": 9000,
"subnet": "172.16.240.0",
"mask": "255.255.255.0",
"gateway": "172.16.240.253",
"ipPools": [
{
"start": "172.16.240.5",
"end": "172.16.240.100"
}
]
},
{
"type": "VMOTION",
"vlanId": 2236,
"mtu": 9000,
"subnet": "172.16.236.0",
"mask": "255.255.255.0",
"gateway": "172.16.236.253",
"ipPools": [
{
"start": "172.16.236.5",
"end": "172.16.236.100"
}
]
}
]
}

So first off lets get a list of current Network Pools. To do this run the following cmdlet:

Get-VCFNetworkPool

As expected this returns a single network pool.

So to create a new network pool using the json you created earlier run the following:

New-VCFNetworkPool -json .\SampleJSON\NetworkPool\addNetworkPoolSpec.json

Now running Get-VCFNetworkPool should display 2 Network Pools

Commission Hosts

Now that you have a network pool you can commission hosts and associate them with the network pool. For this you need the following json

TIP: For this json you need the network pool name & ID. These were returned when the pool was created and also by Get-VCFNetworkPool


[
{
"fqdn": "sfo01w01esx01.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"
},
{
"fqdn": "sfo01w01esx02.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"

},
{
"fqdn": "sfo01w01esx03.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"
},
{
"fqdn": "sfo01w01esx04.sfo01.rainpole.local",
"username": "root",
"storageType": "VSAN",
"password": "VMw@re1!",
"networkPoolName": "sfo01w01-cl01",
"networkPoolId": "afd314f6-f31d-4ad4-8943-0ecb35c044b9"
}
]

So to commission the 4 new hosts into my VCF inventory i simply run

Commission-VCFHost -json .\SampleJSON\Host\commissionHosts.json

TIP: This returns a task id, which you can monitor by running the following until status=Successful:

Get-VCFTask -id b93e2bc7-627b-4f7c-980b-c12b3497c4ea

Create a Workload Domain

Once the commission hosts task is complete you can then create a workload domain using those hosts. Creating a workload domain also requires a json file. For this you need the id’s of the hosts that you want to use. In VCF hosts that are available to be used in a workload domain have a status of UNASSIGNED_USEABLE so to find the id’s of the hosts you want to add run the following

TIP: Filter the results by adding | select fqdn,id

Get-VCFHost -Status UNASSIGNED_USEABLE | select fqdn,id

This returns the ids you need for creating the workload domain. Here is the Workload domain json. (Replace ESXi licence (AAAAA), vSAN licence (BBBBB) & NSX-V licence (CCCCC) with your keys)


{
"domainName" : "PowerVCF",
"vcenterSpec" : {
"name" : "sfo01w01vc01",
"networkDetailsSpec" : {
"ipAddress" : "172.16.225.64",
"dnsName" : "sfo01w01vc01.sfo01.rainpole.local",
"gateway" : "172.16.225.1",
"subnetMask" : "255.255.255.0"
},
"rootPassword" : "VMw@re1!",
"datacenterName" : "PowerVCF-DC"
},
"computeSpec" : {
"clusterSpecs" : [ {
"name" : "Cluster1",
"hostSpecs" : [ {
"id" : "dd2ec05f-39e1-464e-83f1-1349a0dcf723",
"license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
"hostNetworkSpec" : {
"vmNics" : [ {
"id" : "vmnic0",
"vdsName" : "sfo01w01vds01"
}, {
"id" : "vmnic1",
"vdsName" : "sfo01w01vds01"
} ]
}
}, {
"id" : "809b25e8-1db6-464b-b310-97f581c56da5",
"license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
"hostNetworkSpec" : {
"vmNics" : [ {
"id" : "vmnic0",
"vdsName" : "sfo01w01vds01"
}, {
"id" : "vmnic1",
"vdsName" : "sfo01w01vds01"
} ]
}
}, {
"id" : "5d3eea32-6464-4ae6-9866-932fb926a5f1",
"license":"AAAAA-AAAAA-AAAAA-AAAAA-AAAAA",
"hostNetworkSpec" : {
"vmNics" : [ {
"id" : "vmnic0",
"vdsName" : "sfo01w01vds01"
}, {
"id" : "vmnic1",
"vdsName" : "sfo01w01vds01"
} ]
}
} ],
"datastoreSpec" : {
"vsanDatastoreSpec" : {
"failuresToTolerate" : 1,
"licenseKey" : "BBBBB-BBBBB-BBBBB-BBBBB-BBBBB",
"datastoreName" : "sfo01w01vsan01"
}
},
"networkSpec" : {
"vdsSpecs" : [ {
"name" : "sfo01w01vds01",
"portGroupSpecs" : [ {
"name" : "sfo01w01vds01-Mgmt",
"transportType" : "MANAGEMENT"
}, {
"name" : "sfo01w01vds01-VSAN",
"transportType" : "VSAN"
}, {
"name" : "sfo01w01vds01-vMotion",
"transportType" : "VMOTION"
} ]
} ],
"nsxClusterSpec" : {
"nsxVClusterSpec" : {
"vlanId" : 2237,
"vdsNameForVxlanConfig" : "sfo01w01vds01"
}
}
}
} ]
},
"nsxVSpec" : {
"nsxManagerSpec" : {
"name" : "sfo01w01nsx01",
"networkDetailsSpec" : {
"ipAddress" : "172.16.225.66",
"dnsName" : "sfo01w01nsx01.sfo01.rainpole.local",
"gateway" : "172.16.225.1",
"subnetMask" : "255.255.255.0"
}
},
"nsxVControllerSpec" : {
"nsxControllerIps" : [ "172.16.235.121", "172.16.235.122", "172.16.235.123" ],
"nsxControllerPassword" : "VMw@re123456!",
"nsxControllerGateway" : "172.16.235.1",
"nsxControllerSubnetMask" : "255.255.255.0"
},
"licenseKey" : "CCCCC-CCCCC-CCCCC-CCCCC-CCCCC",
"nsxManagerAdminPassword" : "VMw@re1!",
"nsxManagerEnablePassword" : "VMw@re1!"
}
}

To create the workload domain run the following:

New-VCFWorkloadDomain -json .\SampleJSON\WorkloadDomain\workloadDomainSpec-NSX-V.json

This will return a Task ID. Monitor the workload domain creation by running the following

Get-VCFTask -id b93e2bc7-627b-4f7c-980b-c12b3497c4ea

And that should be it. If you’ve gotten all your json details correct you should have a fully functioning NSX-V workload domain without using the UI!