VMware Cloud Foundation Bringup With Signed Certs on ESXi Hosts

Traditionally VMware Cloud Foundation (VCF) has followed the hybrid approach when it comes to SSL certificate management. Hybrid mode essentially means using CA signed certs for the vCenter Server machineSSL cert, and VMCA signed certs for the solution user certs. In this mode, ESXi host certs are VMCA managed also. You then have the option to integrate with an external Microsoft CA or continue to use VMCA for all certs. If you decide to integrate with a Microsoft CA, ESXi host certs remain VMCA managed. This is not always ideal as some customers require all components on the network to be signed by a known & trusted CA. Up until the recent 4.1 VMware Cloud Foundation (VCF) release it was not possible to use custom CA signed certs on your ESXi hosts, as hybrid mode would overwrite your CA signed ESXi certs with VMCA signed certs. There is a great blog post here on how to manually enable CA signed certs here but with VCF 4.1 it is now supported to do this via the API during bringup. The procedure is as follows:


  1. Install the ESXi hosts that will be used for bringup with the ESXi version on the Bill Of Materials for 4.1
  2. Install your custom CA signed certs on each host that will be used for the management domain
    1. Log in to the ESXi Shell, either directly from the DCUI or from an SSH client, as a user with administrator privileges.
    2. In the directory /etc/vmware/ssl, rename the existing certificates using the following commands.
      mv rui.crt orig.rui.crt 
      mv rui.key orig.rui.key
    3. Copy the certificates that you want to use to /etc/vmware/ssl.
    4. Rename the new certificate and key to rui.crt and rui.key.
    5. Restart the host management agents by running the following commands
 /etc/init.d/hostd restart /etc/init.d/vpxa restart 

Repeat the above steps for all management domain hosts

To ensure that SDDC Manager is aware that you are using custom certs you need to add a flag in the bringup json along with the PEM encoded signing chain certificate, so that it is added to the SDDC Manager keystore. This will ensure the certificates are trusted. The API guide for 4.1 provides an example json spec here. Pay particular attention to this section

securitySpec" : {
"esxiCertsMode" : "One among:Custom, VMCA",
"rootCaCerts" : [ {
"alias" : "string",
"certChain" : [ "string" ]
} ]

So to enable support for signed certs you would set this section as follows (Substituting your signing CA chain)

 securitySpec" : { 
"esxiCertsMode" : "Custom", 
"rootCaCerts" : [ { 
"alias" : "Rainpole-CA", 
"certChain" : [ "-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----" ] } ] } 



You can then follow the steps outlined in the API guide to deploy the management domain using the Cloud Builder API. Note that once custom mode is enabled, all future workload domains that you create must also use signed certs.

Announcing VMware Skyline Support for VMware Cloud Foundation 4.1

VMware Skyline™ is a proactive support service aligned with VMware Global Support Services. VMware Skyline automatically and securely collects, aggregates, and analyzes product usage data which proactively identifies potential problems and helps VMware Technical Support Engineers improve the resolution time.

With the release of VMware Skyline collector 2.6, it is now supported to add VMware Cloud Foundation awareness to Skyline advisor to ensure that Skyline findings and recommendations take into account, and do not violate the VMware Cloud Foundation design or Bill Of Materials (BOM). To enable this integration you add the SDDC Manager from each Cloud Foundation instance. When a vCenter Server, NSX-T Manager, & vRealize Operations Manager from that VMware Cloud Foundation instance is added they are automatically associated with the SDDC Manager and tagged for VCF based recommendations in Skyline Advisor.


To add Cloud Foundation to Skyline you need to do the following

Add a user in SDDC Manager and assign the VMware Cloud Foundation Viewer role.

Configure the VMware Skyline Collector to add the SDDC Manager instance by entering the FQDN & credentials for the above user

Once added SDDC Manager will show in the collector inventory view


Logging into Skyline Advisor in VMware Cloud Services you now see VMware Cloud Foundation listed as part of the inventory on the dashboard.

Navigating to the Inventory tab enables you to expand the VMware Cloud Foundation view to see the associated Cloud Foundation inventory

  1. VMware Cloud Foundation
  2. SDDC Manager
  3. Management Domain
  4. Workload Domain(s)
  5. vRealize Operations Manager


Skyline findings and recommendations for these associated inventory items will now be surfaced as solution-based Proactive Findings.

Get started with Skyline here https://docs.vmware.com/en/VMware-Skyline/index.html

PowerVCF 2.0 Authentication Changes

One of the many major enhancements in VMware Cloud Foundation 4.0 is a switch from basic authentication to token based authentication for the VCF API.

Basic authentication is a header field in the form of Authorization: Basic <credentials>, where credentials is the base64 encoding of a username and password. The credentials are not encrypted, therefore Basic Authentication is not the industry standard for API authentication.

VCF 4.0 has moved to using token based authentication (JWT Tokens to be exact) for securing the API. The token implementation is as follows:

  1. An authorized user executes a POST API call to /v1/tokens
  2. The response contains an access token and a refresh token
    1. The access token is valid for 1 hour
      1. The access token is passed in every API call header in the form of Authorization: Bearer <access token>
    2. The refresh token is valid for 24 hours
      1. The refresh token is used to request a new access token once it has expired

PowerVCF 2.0 abstracts all of this in the following way:

  • An authorized user connects to SDDC Manager to request the tokens by running:

Connect-VCFManager -fqdn sfo-vcf01.sfo.rainpole.io -username svc-vcf-api@rainpole.io -password VMw@re1!

  • The access & refresh tokens are stored in memory and used when running subsequent API calls. As each API call is executed PowerVCF checks the expiry of the access token. If the access token is about to expire, it uses the refresh token to request a new access token and proceeds with the API call. So the user does not need to worry about token management.

We have also introduced roles that can be assigned to users. Initially we have ADMIN & OPERATOR, with more roles planned for a future release.

ADMIN = Full Administrator Access to all APIs

OPERATOR = All Access except Password Management, User Management, Backup Management

To request an API token you must have a user account that is assigned either the ADMIN or OPERATOR role in SDDC Manager. The default administrator@vsphere.local user is assigned the ADMIN role during bringup but it is advisable to add additional users for performing day to day tasks.

Once you have a user added you can then authenticate with SDDC Manager to retrieve your access & refresh tokens.

Tip: You can connect using the administrator@vsphere.local user to add new users using PowerVCF. You can use the New-VCFUser PowerVCF cmdlet to create the user and assign a role like so:

Connect-VCFManager -fqdn sfo-vcf01.sfo.rainpole.io -username administrator@vsphere.LOCAL -password VMw@re1!

New-VCFUser -user vcf-admin@rainpole.io -role ADMIN

Once your user is configured PowerVCF will do the rest when it comes to managing the API access tokens.


Announcing PowerVCF 2.0

I’m happy to announce the availability of PowerVCF 2.0. This version of PowerVCF is compatible with VMware Cloud Foundation 4.0 and above. Due to some API security enhancements in VCF 4.0 around the use of API tokens for authentication the module has been refactored to leverage access & refresh tokens (more on that here). For that reason if you would like to use PowerVCF for VCF 3.9.x you should continue to use PowerVCF 1.2 .

PowerVCF 2.0 is published to the PowerShell Gallery here https://www.powershellgallery.com/packages/PowerVCF/2.0.0

Whats new in PowerVCF 2.0

Along with a number of new or modified cmdlets the following enhancements have been made:

  • Grouped cmdlets based on order of API documentation
  • Code hygiene

The following table provides a detailed breakdown of all the changes for this release. Thanks to my colleague @GaryJBlake for doing most of the work on this release and for pulling this list together!

Category cmdlet Name Description Comment
Backup and Restore Start-VCFRestore Starts the restore process of SDDC Manager NEW
Backup and Restore Get-VCFRestoreTasks Gets the status of the restore process NEW
Connectivity Connect-VCFManager Create authentication header for SDDC Manager appliance UPDATED – Support the new token / bearer authentication model and basicAuth switch for restore process
Connectivity Connect-CloudBuilder Create authentication header for Cloud Builder appliance NEW
Certificates Get-VCFCertificateAuthority Get Certificate Authority information UPDATED – Added support for getting the details by id
Certificates Remove-VCFCertificateAuthority Deletes Certificate Authority configuration NEW
Certificates Get-VCFCertificate View certificate of all the resources in a domain UPDATED – Added support for get certificate details by resource
Credentials Get-VCFCredential Get the credentials UPDATED- Added support for getting the details by id
Credentials Stop-VCFCredentialTask Cancels a failed update or rotate passwords task RENAMED – From Cancel-VCFCredentialTask
Credentials Restart-VCFCredentialTask Retry a failed rotate/update passwords task RENAMED – From Retry-VCFCredentialTask
Hosts Commission-VCFHost Commissions a list of hosts UPDATED – Added support for validating the input spec for host operations (-validate switch)
NSX-T Edge Clusters Get-VCFEdgeCluster Get an Edge Cluster NEW
NSX-T Edge Clusters New-VCFEdgeCluster creates an NSX-T edge cluster NEW
Personalities Get-VCFPersonality Get the vSphere Lifecycle Manager Personalities NEW
SDDC (Cloud Builder) Get-CloudBuilderSDDC Retrieve all SDDCs NEW
SDDC (Cloud Builder) Start-CloudBuilderSDDC Create SDDC NEW
SDDC (Cloud Builder) Restart-CloudBuilderSDDC Retry failed SDDC creation NEW
SDDC (Cloud Builder) Get-CloudBuilderSDDCValidation Get all SDDC specification validations NEW
SDDC (Cloud Builder) Start-CloudBuilderSDDCValidation Validate SDDC specification before creation NEW
SDDC (Cloud Builder) Stop-CloudBuilderSDDCValidation Cancel SDDC specification validation NEW
SDDC (Cloud Builder) Restart-CloudBuilderSDDCValidation Retry SDDC validation NEW
System Prechecks Start-VCFSystemPrecheck Perform System Precheck RENAMED – From Start-PreCheckVCFSystem
System Prechecks Get-VCFSystemPrecheckTask Get System Precheck Task RENAMED – From Get-PreCheckVCFSystemTask
Tasks Restart-VCFTask Retry a previously failed task RENAMED – From Retry-VCFTask
Users Get-VCFRole Get all roles NEW
Users Get-VCFUser Get all Users NEW
Users New-VCFUser Adds a new user NEW
Users New-VCFServiceUser Adds a new service user NEW
Users Delete-User Deletes a user NEW
vRealize Suite Lifecycle Manager Reset-VCFvRSLCM Redeploy vRealize Suite Lifecycle Manager NEW
vRealize Suite Lifecycle Manager New-VCFvRSLCM Validate the input specification for vRealize Suite Lifecycle Manager deployment UPDATED – Added support for validating the json spec (-validate switch).

PowerShell Script to Configure an NSX-T Load Balancer for the vRealize Suite & Workspace ONE Access

As part of my role in the VMware Hyper-converged Business Unit (HCIBU) I spend a lot of time working with new product versions testing integrations for next-gen VMware Validated Designs and Cloud Foundation. A lot of my focus is on Cloud Operations and Automation (vROPs, vRLI, vRA etc) and consequently I regularly need to deploy environments to perform integration testing. I will typically leverage existing automation where possible and tend to create my own when i find gaps. Once such gap was the ability to use PowerShell to interact with the NSX-T API. For anyone who is familiar with setting up a load balancer for the vRealize Suite in NSX-T – there are a lot of manual clicks required. So i set about creating some PowerShell functions to make it a little less tedious and to speed up getting my environments setup so i could get to the testing faster.

There is comprehensive NSX-T API documentation posted on code.vmware .com that I used to decipher the various API endpoints required to complete the various tasks:

  • Create the Load Balancer
  • Create the Service Monitors
  • Create the Application Profiles
  • Create the Server Pools
  • Create the Virtual Servers

The result is a PowerShell module with a function for each of the above and a corresponding JSON file that is read in for the settings for each function. I have included a sample JSON file to get you started. Just substitute your values.

Note: You must have a Tier-1 & associated segments created. (I’ll add that functionality when i get a chance!)

PowerShell Module, Sample JSON & Script are posted to Github here

Create a multi pNIC VMware Cloud Foundation NSX-V Workload Domain with PowerVCF

Hopefully by now you’ve seen my earlier posts about the new PowerShell module for the VMware Cloud Foundation API. If not i’d suggest reviewing these before reading on

With the release of VMware Cloud Foundation 3.9.1 it is now supported, via the API only, to use more than 2 physical NICs (pNICs) per host. In fact the API now supports up to three vSphere Distributed switches and six physical NICs, providing more flexibility to support high performance use cases and physical traffic separation.

There is a tech note that goes into more detail on the use cases for more than 2 pNICs and it also shows how this works using PostMan but we can also achieve this using PowerVCF.

The workflow using PowerVCF is the same as my earlier example for creating a workload domain. The only difference is the content in the JSON file.

Note: There is a validation API to validate the JSON you are passing before making the submission. PowerVCF dynamically formats the validation JSON as the formatting is slightly different to what you submit to create the workload domain.

To get you started there is a sample JSON file with the required formatting. Here is a snapshot of what it looks like

  "domainName": "PowerVCF",  
  "vcenterSpec": {  
    "name": "sfo01w01vc01",  
    "networkDetailsSpec": {  
       "ipAddress": "",  
       "dnsName": "sfo01w01vc01.sfo01.rainpole.local",  
       "gateway": "",  
       "subnetMask": ""
     "rootPassword": "VMw@re1!",  
     "datacenterName": "PowerVCF-DC"  
   "computeSpec": {  
      "clusterSpecs": [ {  
          "name": "Cluster1",  
          "hostSpecs": [ {  
              "id": "d0693b58-4012-4387-92ed-721cfa709e44",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ]  
            }, {  
              "id": "7006bec4-fccb-49a0-bff6-fd56c807d26a",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ]  
            }, {  
              "id": "cc257a80-e179-4297-bf7e-179a0944bbab",
              "hostNetworkSpec": {  
                 "vmNics": [ {  
                     "id": "vmnic0",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, {  
                     "id": "vmnic1",  
                     "vdsName": "SDDC-Dswitch-Private1"  
                  }, { 
                     "id": "vmnic2",  
                     "vdsName": "SDDC-Dswitch-Private2" 
                  }, {  
                     "id": "vmnic3",  
                     "vdsName": "SDDC-Dswitch-Private2"  
                  } ] 
           } ],     
    "datastoreSpec": {  
        "vsanDatastoreSpec": {  
            "failuresToTolerate": 1,  
            "licenseKey": "BBBBB-BBBBB-BBBBB-BBBBB-BBBBB",
            "datastoreName": "vSanDatastore" 
     "networkSpec": { 
         "vdsSpecs": [ { 
             "name": "SDDC-Dswitch-Private1", 
             "portGroupSpecs": [ {  
                 "name": "SDDC-DPortGroup-Mgmt", 
                 "transportType": "MANAGEMENT" 
             }, { 
                 "name": "SDDC-DPortGroup-VSAN",  
                 "transportType": "VSAN" 
             }, {  
                 "name": "SDDC-DPortGroup-vMotion", 
                 "transportType": "VMOTION" 
             } ] 
             "name": "SDDC-Dswitch-Private2", 
             "portGroupSpecs": [ { 
                "name": "SDDC-DPortGroup-Public", 
                "transportType": "PUBLIC"  } ] 
        "nsxClusterSpec": { 
           "nsxVClusterSpec": {  
              "vlanId": 2237,  
              "vdsNameForVxlanConfig": "SDDC-Dswitch-Private1"  
      } ] 
  "nsxVSpec" : {
    "nsxManagerSpec" : {
      "name" : "sfo01w01nsx01",
      "networkDetailsSpec" : {
        "ipAddress" : "",
        "dnsName" : "sfo01w01nsx01.sfo01.rainpole.local",
        "gateway" : "",
        "subnetMask" : ""
    "nsxVControllerSpec" : {
      "nsxControllerIps" : [ "", "", "" ],
      "nsxControllerPassword" : "VMw@re123456!",
      "nsxControllerGateway" : "",
      "nsxControllerSubnetMask" : ""
    "nsxManagerAdminPassword" : "VMw@re1!",
    "nsxManagerEnablePassword" : "VMw@re1!"

You can see that the magic happens in the hostNetworkSpec section where you map each vmnic to a vdsName

<p>"hostNetworkSpec": { "vmNics": [ { "id": "vmnic0", "vdsName": "SDDC-Dswitch-Private1" }, { "id": "vmnic1", "vdsName": "SDDC-Dswitch-Private1" }, { "id": "vmnic2", "vdsName": "SDDC-Dswitch-Private2" }, { "id": "vmnic3", "vdsName": "SDDC-Dswitch-Private2" } ] }</p>

So please try it out and let us know how it goes!

Introducing PowerVCF – A PowerShell Module for the VMware Cloud Foundation API

Its been a while since I’ve posted something so I thought it was about time! Since joining VMware a year ago I’ve been heads down drinking from the firehose, learning from a phenomenal team and generally keeping very busy. More recently I’ve been playing a lot with VMware Cloud Foundation (VCF). A recent release (3.8) introduced a public API and I started getting field questions on how to leverage it so I started digging. The API has been expanded in 3.9. It is based on the OpenAPI standard (formerly Swagger) and can be accessed through the developer center in the SDDC Manager UI or via code.vmware.com

Now I’m not a developer so I fell back on Postman to do some initial testing. I like Postman as it dumbs it down for us non-devs 🙂 but I wanted something a little easier to consume so i started a little side project called PowerVCF (hat-tip to the far superior PowerNSX, PowerVRA, PowerVRO)

Basically I wanted to provide a simple, efficient, PowerCLI style experience for consuming the VMware Cloud Foundation public API.


I am delighted to unleash the first iteration of PowerVCF on the community! Creating this has been a great learning experience for me. In the process I’ve improved my PowerShell skills, learned Git, Markdown and have started looking into CI/CD workflows. It’s also my first submission to the PowerShell Gallery.

Reset/Unlock Photon OS root account

From time to time your root account can get locked from either entering the incorrect password or using some automation that uses the wrong password. Here are some quick steps.

Reboot the Photon Appliance

At the Photon OS logo screen press e to edit the grub menu

At the grub menu append the following to the end of the boot loader line to boot into single user mode

rw init=/bin/bash

Press F10 or CTRL+X to continue the boot process

At the prompt type the following to mount the root partition

mount -o remount,rw /

To reset the root password type passwd and enter the new password

If the root account was locked due to x number of failed logon attempts type to following to unlock it

/sbin/pam_tally2 -r -u root

Unmount the partition again

umount /

And reboot

reboot -f

Hopefully you should now be able to log in with your root account!

Verify NTP Configuration on a VMware Photon Appliance

Quick post with the commands required to verify NTP configuration on a VMware Photon OS Appliance

ssh to the appliance as root

To check if the NTP service is up and running you can run this command

systemctl status systemd-timesyncd

If its stopped run this command

systemctl start systemd-timesyncd

To configure NTP servers run the following

vi /etc/systemd/timesyncd.conf

Add timeservers under [Time]

Automate your VMware Validated Design NSX-V Distributed Firewall Configuration

A few weeks back I mentioned on twitter that i was working on automating the VMware Validated Design NSX-V Distributed Firewall Configuration in my lab. (I admit it took longer than i had planned!) Currently this is a manual post deployment step once VMware Cloud Builder has completed the deployment. This will likely be picked up by Cloud Builder in a future release but for now its a manual, and somewhat tedious, but required, step!

Full details on the manual steps required for this configuration can be found here. Please take the time to understand what these rules are doing before implementing them.

So in an effort to make this post configuration step a little less painful i set out to automate it. I’ve played with the NSX-V API in the past and found it much easier to interact with by using PowerNSX, rather than leveraging PostMan and the API directly. PowerNSX is the unofficial, official automation tool for NSX. Hats off to VMware engineers Nick Bradford, Dale Coghlan & Anthony Burke for creating and documenting this tool. Anthony also published a FREE book on Automating NSX for vSphere with PowerNSX. More on that here.

Disclaimer: This script is not officially supported by VMware. Use at your own risk & test in a development/lab environment before using in production.

I’ve posted the script to GitHub here as its a bit lengthy! There may be a more efficient way to do some parts of it and if anyone wants to contribute please feel free!

As with a lot of the scripts i create it is menu based and has 2 main options:

  1. Create DFW exclusions, IP Sets & Security Groups
  2. Create DFW Rules

The reason i split it into 2 distinct operations is to allow you to inspect the exclusion list, IP Sets & Security Groups before creating the firewall rules. This will ensure that you dont lock yourself out of vCenter by creating an incorrect rule.

Required Software

  • PowerCli
    • The script will check for PowerCli and if not found will attempt to install the latest version from the PowerShell Gallery
    • Currently tested on Windows only
    • If you dont have internet access you can manually install PowerCli by opening a PowerShell console as administrator and running:
    • Find-Module -Name VMware.PowerCLI | Install-Module
  • PowerNSX
    • The script will check for PowerNSX and if not found will attempt to install the latest version from the PowerShell Gallery
    • Currently tested on Windows only
    • If you dont have internet access you can manually install PowerNSX by opening a PowerShell console as administrator and running:
    • Find-Module -Name PowerNSX | Install-Module

Required Variables

Before you can run the script you need to edit the User Variables to provide the following:

  • Target vCenter details
    • Required to establish a PowerCli Connection with vCenter Server. This is used when updating the DFW exclusion list
  • Target NSX Manager details
    • Required to establish a connection with NSX manager to configure the DFW
  • IP Addresses for the various SDDC components

Hopefully you will find this useful!