My name is Brian O’Connell. I work for VMware as part of the Cloud Infrastructure Business Group working on VMware Cloud Foundation (VCF) architecture design. I am an IT geek at heart and love playing with new technologies. I started my career in an IT services role as a Systems administrator for SMB’s and moved to EMC in 2010 where I worked for the EMC vLab team developing content for the ever expanding vLab cloud platform. I moved into working with the EMC Cloud Engineering Team primarily focused on the EMC Enterprise Hybrid Cloud Solution (EHC). I am also a husband to a beautiful wife, father to 2 beautiful daughters and a musician (or failed rock star depending on how you look at it!). Hope you fins my content useful!
I’ve recently been doing a lot of work with VMware Site Recovery Manager (SRM) and vSphere Replication (vSR) with VMware Cloud Foundation. Earlier this year we (Ken Gould & I) published an early access design for Site Protection & Recovery for VMware Cloud Foundation 4.2. We have been working to refresh & enhance this design for a new release. Part of this effort includes trying to add some automation to assist with the manual steps to speed up time to deploy. SRM & vSR do not have publicly documented VAMI APIs so we set about trying to automate the configuration with a little bit of reverse engineering.
As with most APIs, whether public or private, you must authenticate before you can run an API workflow, so the first task is figuring out how the authentication to perform a workflow works. Typically, if you hit F12 in your browser you will get a developer console that exposes what goes on behind the scenes in a browser session. So to inspect the process, use the browser to perform a manual login, and review the header & response tabs in the developer view. This exposes the Request URL to use, the method (POST) and the required headers (accept: application/json)
The Response tab shows a sessionId which can be used for further configuration API calls in the headers as dr.config.service.sessionid
So with the above information you can use an API client like Postman to retrieve a sessionId with the URL & headers like this
And your VAMI admin user and password in JSON format in the body payload
You can also use the information to retrieve a sessionId using PowerShell
Within a VMware Cloud Foundation instance, SDDC Manager is used to manage the lifecycle of passwords (or credentials). While we provide the ability to rotate (either scheduled or manually) currently there is no easy way to check when a particular password is due to expire, which can lead to appliance root passwords expiring, which will cause all sorts of issues. The ability to monitor expiry is something that is being worked on, but as a stop gap I put together the script below which leverages PowerVCF and also a currently undocumented API for validating credentials.
The script has a function called Get-VCFPasswordExpiry that accepts the following parameters
-fqdn (FQDN of the SDDC Manager)
-username (SDDC Manager Username – Must have the ADMIN role)
-password (SDDC Manager password)
-resourceType (Optional parameter to specify a resourceType. If not passed, all resources will be checked. If passed (e.g. VCENTER) then only that resourceType will be checked. Supported resource types are
PowerVCF is a requirement. If you dont already have it run the following
Install-Module -Name PowerVCF
The code takes a while to run as it needs to do the following to check password expiry
Connect to SDDC Manager to retrieve an API token
Retrieve a list of all credentials
Using the resourceID of each credential
Perform a credential validation
Wait for the validation to complete
Parse the results for the expiry details
Add all the results to an array and present in a table (Kudos to Ken Gould for assistance with the presentation of this piece!)
In this example script I am returning all non SERVICE user accounts regardless of expiry (SERVICE account passwords are system managed). You could get more granular by adding something like this to only display accounts with passwords due to expire in less than 14 days
if ($validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry -lt 14) {
Write-Output "Password for username $($validationTaskResponse.validationChecks.username) expires in $($validationTaskResponse.validationChecks.passwordDetails.numberOfDaysToExpiry) days"
}
PowerVCF 2.1.1 is now available on the PowerShell Gallery here or from GitHub here. This release includes support for VMware Cloud Foundation 4.1 with some new and updated cmdlets. Highlights below
Category
cmdlet Name
Description
Comment
Users and Groups
Get-VCFCredential
Retrieves a list of credentials.
UPDATE: Added support for the vRealize Suite credentials
Bundles
Start-VCFBundleUpload
Starts upload of bundle to SDDC Manager
UPDATE: Allows the import of a bundle based on offline download.
Federation
New-VCFFederationInvite
Invite new member to VCF Federation
UPDATE: Added support to specify if the new system is a MEMBER or CONTROLLER.
SDDC
Start-CloudBuilderSDDCValidation
Starts validation on VMware Cloud Builder
UPDATE: Added support for individual validation tasks.
Workspace ONE Access
Get-VCFWSA
Get details of the existing Workspace ONE Access
NEW
vRealize Automation
Get-VCFvRA
Get details of the the existing vRealize Automation
NEW
vRealize Operations
Get-VCFvROPs
Get details of the existing vRealize Operations Manager
NEW
vRealize Operations
Set-VCFvROPs
Connect or disconnect Workload Domains to vRealize Operations Manager
NEW
vRealize Log Insight
Get-VCFvRLI
Get details of the existing vRealize Log Insight
NEW
vRealize Log Insight
Set-VCFvRLI
Connect or disconnect Workload Domains to vRealize Log Insight
NEW
vRealize Suite Lifecycle
Get-VCFvRSLCM
Get details of the existing vRealize Suite Lifecycle Manager
UPDATE: Fixed an issue with the API URI and addressed response output
Traditionally VMware Cloud Foundation (VCF) has followed the hybrid approach when it comes to SSL certificate management. Hybrid mode essentially means using CA signed certs for the vCenter Server machineSSL cert, and VMCA signed certs for the solution user certs. In this mode, ESXi host certs are VMCA managed also. You then have the option to integrate with an external Microsoft CA or continue to use VMCA for all certs. If you decide to integrate with a Microsoft CA, ESXi host certs remain VMCA managed. This is not always ideal as some customers require all components on the network to be signed by a known & trusted CA. Up until the recent 4.1 VMware Cloud Foundation (VCF) release it was not possible to use custom CA signed certs on your ESXi hosts, as hybrid mode would overwrite your CA signed ESXi certs with VMCA signed certs. There is a great blog post here on how to manually enable CA signed certs here but with VCF 4.1 it is now supported to do this via the API during bringup. The procedure is as follows:
Install the ESXi hosts that will be used for bringup with the ESXi version on the Bill Of Materials for 4.1
Install your custom CA signed certs on each host that will be used for the management domain
Log in to the ESXi Shell, either directly from the DCUI or from an SSH client, as a user with administrator privileges.
In the directory /etc/vmware/ssl, rename the existing certificates using the following commands.
mv rui.crt orig.rui.crt
mv rui.key orig.rui.key
Copy the certificates that you want to use to /etc/vmware/ssl.
Rename the new certificate and key to rui.crt and rui.key.
Restart the host management agents by running the following commands
Repeat the above steps for all management domain hosts
To ensure that SDDC Manager is aware that you are using custom certs you need to add a flag in the bringup json along with the PEM encoded signing chain certificate, so that it is added to the SDDC Manager keystore. This will ensure the certificates are trusted. The API guide for 4.1 provides an example json spec here. Pay particular attention to this section
You can then follow the steps outlined in the API guide to deploy the management domain using the Cloud Builder API. Note that once custom mode is enabled, all future workload domains that you create must also use signed certs.
VMware Skyline™ is a proactive support service aligned with VMware Global Support Services. VMware Skyline automatically and securely collects, aggregates, and analyzes product usage data which proactively identifies potential problems and helps VMware Technical Support Engineers improve the resolution time.
With the release of VMware Skyline collector 2.6, it is now supported to add VMware Cloud Foundation awareness to Skyline advisor to ensure that Skyline findings and recommendations take into account, and do not violate the VMware Cloud Foundation design or Bill Of Materials (BOM). To enable this integration you add the SDDC Manager from each Cloud Foundation instance. When a vCenter Server, NSX-T Manager, & vRealize Operations Manager from that VMware Cloud Foundation instance is added they are automatically associated with the SDDC Manager and tagged for VCF based recommendations in Skyline Advisor.
To add Cloud Foundation to Skyline you need to do the following
Add a user in SDDC Manager and assign the VMware Cloud Foundation Viewer role.
Configure the VMware Skyline Collector to add the SDDC Manager instance by entering the FQDN & credentials for the above user
Once added SDDC Manager will show in the collector inventory view
Logging into Skyline Advisor in VMware Cloud Services you now see VMware Cloud Foundation listed as part of the inventory on the dashboard.
Navigating to the Inventory tab enables you to expand the VMware Cloud Foundation view to see the associated Cloud Foundation inventory
VMware Cloud Foundation
SDDC Manager
Management Domain
Workload Domain(s)
vRealize Operations Manager
Skyline findings and recommendations for these associated inventory items will now be surfaced as solution-based Proactive Findings.
From time to time we all need to look at logs, whether its a failed operation or to trace who did what when. In VMware Cloud Foundation there are many different logs, each one serving a different purpose. Its not always clear which log you should look at for each operation so here is a useful reference table.
One of the many major enhancements in VMware Cloud Foundation 4.0 is a switch from basic authentication to token based authentication for the VCF API.
Basic authentication is a header field in the form of Authorization: Basic <credentials>, where credentials is the base64 encoding of a username and password. The credentials are not encrypted, therefore Basic Authentication is not the industry standard for API authentication.
VCF 4.0 has moved to using token based authentication (JWT Tokens to be exact) for securing the API. The token implementation is as follows:
An authorized user executes a POST API call to /v1/tokens
The response contains an access token and a refresh token
The access token is valid for 1 hour
The access token is passed in every API call header in the form of Authorization: Bearer <access token>
The refresh token is valid for 24 hours
The refresh token is used to request a new access token once it has expired
PowerVCF 2.0 abstracts all of this in the following way:
An authorized user connects to SDDC Manager to request the tokens by running:
The access & refresh tokens are stored in memory and used when running subsequent API calls. As each API call is executed PowerVCF checks the expiry of the access token. If the access token is about to expire, it uses the refresh token to request a new access token and proceeds with the API call. So the user does not need to worry about token management.
We have also introduced roles that can be assigned to users. Initially we have ADMIN & OPERATOR, with more roles planned for a future release.
ADMIN = Full Administrator Access to all APIs
OPERATOR = All Access except Password Management, User Management, Backup Management
To request an API token you must have a user account that is assigned either the ADMIN or OPERATOR role in SDDC Manager. The default administrator@vsphere.local user is assigned the ADMIN role during bringup but it is advisable to add additional users for performing day to day tasks.
Once you have a user added you can then authenticate with SDDC Manager to retrieve your access & refresh tokens.
Tip: You can connect using the administrator@vsphere.local user to add new users using PowerVCF. You can use the New-VCFUser PowerVCF cmdlet to create the user and assign a role like so:
I’m happy to announce the availability of PowerVCF 2.0. This version of PowerVCF is compatible with VMware Cloud Foundation 4.0 and above. Due to some API security enhancements in VCF 4.0 around the use of API tokens for authentication the module has been refactored to leverage access & refresh tokens (more on that here). For that reason if you would like to use PowerVCF for VCF 3.9.x you should continue to use PowerVCF 1.2 .
Along with a number of new or modified cmdlets the following enhancements have been made:
Grouped cmdlets based on order of API documentation
Code hygiene
The following table provides a detailed breakdown of all the changes for this release. Thanks to my colleague @GaryJBlake for doing most of the work on this release and for pulling this list together!
Category
cmdlet Name
Description
Comment
Backup and Restore
Start-VCFRestore
Starts the restore process of SDDC Manager
NEW
Backup and Restore
Get-VCFRestoreTasks
Gets the status of the restore process
NEW
Connectivity
Connect-VCFManager
Create authentication header for SDDC Manager appliance
UPDATED – Support the new token / bearer authentication model and basicAuth switch for restore process
Connectivity
Connect-CloudBuilder
Create authentication header for Cloud Builder appliance
NEW
Certificates
Get-VCFCertificateAuthority
Get Certificate Authority information
UPDATED – Added support for getting the details by id
Certificates
Remove-VCFCertificateAuthority
Deletes Certificate Authority configuration
NEW
Certificates
Get-VCFCertificate
View certificate of all the resources in a domain
UPDATED – Added support for get certificate details by resource
Credentials
Get-VCFCredential
Get the credentials
UPDATED- Added support for getting the details by id
Credentials
Stop-VCFCredentialTask
Cancels a failed update or rotate passwords task
RENAMED – From Cancel-VCFCredentialTask
Credentials
Restart-VCFCredentialTask
Retry a failed rotate/update passwords task
RENAMED – From Retry-VCFCredentialTask
Hosts
Commission-VCFHost
Commissions a list of hosts
UPDATED – Added support for validating the input spec for host operations (-validate switch)
NSX-T Edge Clusters
Get-VCFEdgeCluster
Get an Edge Cluster
NEW
NSX-T Edge Clusters
New-VCFEdgeCluster
creates an NSX-T edge cluster
NEW
Personalities
Get-VCFPersonality
Get the vSphere Lifecycle Manager Personalities
NEW
SDDC (Cloud Builder)
Get-CloudBuilderSDDC
Retrieve all SDDCs
NEW
SDDC (Cloud Builder)
Start-CloudBuilderSDDC
Create SDDC
NEW
SDDC (Cloud Builder)
Restart-CloudBuilderSDDC
Retry failed SDDC creation
NEW
SDDC (Cloud Builder)
Get-CloudBuilderSDDCValidation
Get all SDDC specification validations
NEW
SDDC (Cloud Builder)
Start-CloudBuilderSDDCValidation
Validate SDDC specification before creation
NEW
SDDC (Cloud Builder)
Stop-CloudBuilderSDDCValidation
Cancel SDDC specification validation
NEW
SDDC (Cloud Builder)
Restart-CloudBuilderSDDCValidation
Retry SDDC validation
NEW
System Prechecks
Start-VCFSystemPrecheck
Perform System Precheck
RENAMED – From Start-PreCheckVCFSystem
System Prechecks
Get-VCFSystemPrecheckTask
Get System Precheck Task
RENAMED – From Get-PreCheckVCFSystemTask
Tasks
Restart-VCFTask
Retry a previously failed task
RENAMED – From Retry-VCFTask
Users
Get-VCFRole
Get all roles
NEW
Users
Get-VCFUser
Get all Users
NEW
Users
New-VCFUser
Adds a new user
NEW
Users
New-VCFServiceUser
Adds a new service user
NEW
Users
Delete-User
Deletes a user
NEW
vRealize Suite Lifecycle Manager
Reset-VCFvRSLCM
Redeploy vRealize Suite Lifecycle Manager
NEW
vRealize Suite Lifecycle Manager
New-VCFvRSLCM
Validate the input specification for vRealize Suite Lifecycle Manager deployment
UPDATED – Added support for validating the json spec (-validate switch).
As part of my role in the VMware Hyper-converged Business Unit (HCIBU) I spend a lot of time working with new product versions testing integrations for next-gen VMware Validated Designs and Cloud Foundation. A lot of my focus is on Cloud Operations and Automation (vROPs, vRLI, vRA etc) and consequently I regularly need to deploy environments to perform integration testing. I will typically leverage existing automation where possible and tend to create my own when i find gaps. Once such gap was the ability to use PowerShell to interact with the NSX-T API. For anyone who is familiar with setting up a load balancer for the vRealize Suite in NSX-T – there are a lot of manual clicks required. So i set about creating some PowerShell functions to make it a little less tedious and to speed up getting my environments setup so i could get to the testing faster.
There is comprehensive NSX-T API documentation posted on code.vmware .com that I used to decipher the various API endpoints required to complete the various tasks:
Create the Load Balancer
Create the Service Monitors
Create the Application Profiles
Create the Server Pools
Create the Virtual Servers
The result is a PowerShell module with a function for each of the above and a corresponding JSON file that is read in for the settings for each function. I have included a sample JSON file to get you started. Just substitute your values.
Note: You must have a Tier-1 & associated segments created. (I’ll add that functionality when i get a chance!)
PowerShell Module, Sample JSON & Script are posted to Github here
Hopefully by now you’ve seen my earlier posts about the new PowerShell module for the VMware Cloud Foundation API. If not i’d suggest reviewing these before reading on
With the release of VMware Cloud Foundation 3.9.1 it is now supported, via the API only, to use more than 2 physical NICs (pNICs) per host. In fact the API now supports up to three vSphere Distributed switches and six physical NICs, providing more flexibility to support high performance use cases and physical traffic separation.
There is a tech note that goes into more detail on the use cases for more than 2 pNICs and it also shows how this works using PostMan but we can also achieve this using PowerVCF.
The workflow using PowerVCF is the same as my earlier example for creating a workload domain. The only difference is the content in the JSON file.
Note: There is a validation API to validate the JSON you are passing before making the submission. PowerVCF dynamically formats the validation JSON as the formatting is slightly different to what you submit to create the workload domain.
To get you started there is a sample JSON file with the required formatting. Here is a snapshot of what it looks like