Upgrading VCF 5.2 to 9.0 – Part 9 – Deactivate Enhanced Link Mode (ELM)

The final part of this series on Upgrading VCF 5.2 to 9.0 is to Deactivate Enhanced Link Mode (ELM) on the VCF instance that we have upgraded. ELM has been around forever, but with VCF 9.0, ELM is now deprecated. You can no longer deploy vCenter instances in an ELM ring. VCF 9.0 introduces a new concept of vCenter linking, which, along with VCF SSO, enabled by VCF Identity Broker (VIDB), replaces the functionality previously provided by ELM. I will cover vCenter linking and VCF SSO in a later post, but before you can take advantage of VCF SSO, you must first deactivate ELM.

Deactivating ELM means each vCenter in the ELM ring now becomes a standalone vCenter with its own isolated SSO domain and its is an all or nothing operation, meaning you cannot selectively remove one vCenter from the ELM ring. Once you perform the operation, all vCenters will be removed from the ELM ring, and all vCenters will use their own instance of vsphere.local.

Important Note: Taking offline snapshots of all vCenter instances in the ELM ring is recommended in case you need to revert.

To deactivate ELM, we will use the SDDC Manager API.

Browse to SDDC manager and click on Developer Center
Navigate to Domains and under GET /v1/domains click Execute.
Expand the response and locate the id for the management domain

Expand POST /v1/domains/{id}/validations , enter the management domain id in the id field and the following json in the body, and click Execute.

{
  "breakElmSpec": {
      "isReconcileWorkflow": false
  }
}

Expand GET /v1/domains/{id}/validations/{validation_id} and replace {id} with the management domain ID. Replace {validation_id} with the validation ID, from the previous step, and click Execute. Expand the response to ensure it is successful.

To break ELM across all VCF domains in the VCF instance, expand PATCH /v1/domains/{id} , enter the management domain id in the id field and the following json in the body, and click Execute.

{
  "breakElmSpec": {
      "isReconcileWorkflow": false
  }
}

Locate the task ID in the response.

To monitor the task progress, expand GET /v1/tasks/{id} , enter the task id in the id field, and click Execute.

Adding an Existing VCF Instance to a VCF Fleet

This is a scenario that is not covered very well in our current VCF 9.0 docs (I am working to rectify that), where a customer has more than 1 existing VCF 5.x instance and they want to move to VCF 9.0. The upgrade of the first instance and the addition (or upgrade) of VCF Operations & VCF Automation are documented. But how do you go about adding those additional VCF 5.x instances into your new VCF 9.0 fleet? Although it looks like a long post, the process is pretty straightforward.

TL;DR Version

Create an integration in VCF Operations for the new VCF instance.
Update the integration to use a system-managed service account.
Add the new management domain vCenter as a deployment target in fleet management.
Deploy a VCF Operations collector to the new instance.
Update the integration to use the new collector/collector group.

Step By Step Version

First, you should upgrade the additional instances to 9.0. This is not a mandatory step (and sometimes may not be possible for operational reasons), but it will set you up to be able to consume all of the features of VCF 9.0.

Next, create a new VMware Cloud Foundation integration in VCF Operations for the SDDC Manager of the additional VCF instance.

In the VCF Operation UI, navigate to Administration > Integrations, and click Add.

Click VMware Cloud Foundation on the Account Types screen.

Enter a friendly name for the instance and enter the FQDN for SDDC Manager.
To add a new credential, click the + next to credential

Enter a username and password with the ADMIN role in SDDC Manager, and click OK. For now, I am using administrator@vsphere.local as we will change this to a service account later.

Leave the Collector / Group as Default for now. This will also be changed later.
Click Validate Connection and accept the certificate.

Expand Advanced Settings, change Enable Domain Monitoring on Creation to true, and remove the VCF Configuration limits File Name (if it is populated).

To add the connection, click Add.

Acknowledge the Account Monitoring Notification and click OK.

On the Integrations screen, expand VMware Cloud Foundation and expand the new Integration. You will see that the workload domain components are in a stopped status. To start them, click the vertical ellipsis at the VCF instance level and click Start Collecting All.

To move away from using a defined credential to a system managed service account, click the vertical ellipsis at the VCF instance level and click Edit.

On the Cloud Account Information screen, under System Managed Credential, click Activate, and click Save.

You can now safely delete the ADMIN role credential you created earlier. On the Integrations screen, click Credentials.

On the Credentials screen, click the vertical ellipsis next to the credential you created and click Delete.

Click Yes to confirm the delete operation.

Next you need to add the management domain vCenter as a deployment target to allow you to deploy VCF Operations collector(s).

Navigate to Fleet Management > Lifecycle > Settings > Deployment Targets, and click Add Deployment Target.

The Add Deployment Target wizard will find vCenter instances that are configured as a VCF integration and present a list to choose from.

Select the vCenter and click Validate.

Once the validation is successful, click Add.

Monitor the task status until it completes.

To deploy VCF Operations collector(s) to the newly added VCF instance, navigate to Components and click operations in the components list.

Before adding new nodes to the VCf Operations cluster, it is advisable to perform an inventory sync to avoid any drift. Click Trigger Inventory Sync.

On the Trigger Inventory Sync pop-up, click Submit.

Monitor the inventory sync task until it completes and then navigate back to the Add Nodes screen and click Add Node. As you have already performed the Inventory Sync, it can be ignored here. Click Proceed.

On the Infrastructure screen, select the newly added management domain vCenter from the drop down and select the target cluster, network and datastore and click Next.

On the Networks screen, do not make any changes. You will set the network information on the collector directly. Click Next.

On the Components screen, uncheck Take component snapshot, as it is not required when adding collector nodes.

Before you can deploy the collector, you need to add a root user password for the collector to the credentials store. On the Components screen, click Add Password.

Enter the password details and click Add.

Scroll down to Components and click the + to add a component. From the list, select Cloud Proxy. Note: Cloud Proxy is the legacy name for the VCF Operations Collector.

Enter the details for the collector and click the settings icon on the right.

Update the network details to reflect the newly added VCF instance

If additional NTP servers are required, under NTP Settings, click Add New Server. Enter the server details and click Submit.

To change the NTP Server selection, click Edit Server Selection.

On the Select Servers screen, select the correct server(s) and click Next. If required, adjust the server priority and click Finish.

Under Root Password, click Select Root Password and select the root password created earlier.

Under Timezone, select the correct timezone from the dropdown and click Save.

On the Components screen, click Next.

On the Precheck screen, click Run Precheck, and wait for the precheck to complete successfully. Once its complete, click Next.

On the Summary screen, click Submit to deploy the collector(s).

Monitor the deployment task to completion.

On the Integrations screen, expand VMware Cloud Foundation and expand the new Integration. Click the vertical ellipsis at the VCF instance level and click Edit.

Under Collector / Group, select the newly deployed collector, and click Validate Connection.

That takes care of the instance-level setting, which will be applied to any new workload domains. You must repeat the process at a per-workload domain level for any existing domains.

Click the vertical ellipsis at the domain level, and click Edit.

On the vCenter screen, change the Collector / group to the new collector and click Validate Connection. Click Ok on the Test Connection popup and click Save.

Now you should see your new VCF instance using the correct collector/group.

If you are using VCF Automation, your new VCF instance should now be discovered automatically.

Quick Tip: Retrieve vSphere Supervisor Control Plane SSH Password

SSH access to the Supervisor Control Plane VMs uses an auto-generated password. The password for the system user on these VMs needs to be retrieved from associated vCenter.

IMPORTANT: Direct access to the vSphere Supervisor Control Plane VMs should be used with caution and for troubleshooting purposes only.

SSH to the vCenter as the root user, and enter the bash shell by typing shell

Run the following to retrieve the password

root@sfo-w01-vc01 [ ~ ]# /usr/lib/vmware-wcp/decryptK8Pwd.py

You should now be able to SSH to the IP of the vSphere Supervisor Control Plane VM (The IP returned will be the floating IP or FIP) with username root.

Upgrading VCF 5.2 to 9.0 – Part 8 – Deploy VCF Fleet Management Components

VCF 9.0 introduced the concept of VCF fleet, which is defined as:

An environment that is managed by a single set of fleet-level management components – VCF Operations & VCF Automation. A VCF fleet contains one or more VCF Instances and may contain one or more standalone vCenter instances, managed by the VCF Operations instance for the fleet. The management domain of the first VCF Instance in the VCF fleet typically hosts the fleet-level management components.

When deploying a new VCF fleet, you get the option to deploy the fleet-level management components using the VCF installer. Because I am upgrading from VCF 5.2, where I did not have Aria Operations or Aria Automation, I need to deploy new instances of each component (If I had pre-existing instances, they could be upgraded). You can deploy them manually from OVA, however, there is a new SDDC Manager API to automate the process using a JSON payload.

The API can be accessed via the SDDC Manager developer centre, under VCF Management Components.

The JSON payload to deploy VCF Operations (including a collector & the fleet management appliance) and VCF Automation is as follows: (NOTE: This spec is for a simple/single node deployment of the fleet management components where VCF Operations & VCF Automation will be deployed to an NSX Overlay segment, and the VCF Operations collector will be deployed to the management DVPG)

 {
    "vcfOperationsFleetManagementSpec": {
        "hostname": "flt-fm01.rainpole.io",
        "rootUserPassword": "VMw@re1!VMw@re1!",
        "adminUserPassword": "VMw@re1!VMw@re1!",
        "useExistingDeployment": false
    },
    "vcfOperationsSpec": {
        "nodes": [
            {
                "hostname": "flt-ops01a.rainpole.io",
                "rootUserPassword": "VMw@re1!VMw@re1!",
                "type": "master"
            }
        ],
        "useExistingDeployment": false,
        "applianceSize": "medium",
        "adminUserPassword": "VMw@re1!VMw@re1!"
    },
    "vcfOperationsCollectorSpec": {
        "hostname": "sfo-opsc01.sfo.rainpole.io",
        "rootUserPassword": "VMw@re1!VMw@re1!",
        "applianceSize": "small"
    },
    "vcfAutomationSpec": {
        "hostname": "flt-auto01.rainpole.io",
        "adminUserPassword": "VMw@re1!VMw@re1!",
        "useExistingDeployment": false,
        "ipPool": [
            "192.168.11.51",
            "192.168.11.52"
        ],
        "internalClusterCidr": "250.0.0.0/15",
        "vmNamePrefix": "flt-auto01"
    },
    "vcfInstanceName": "San Francisco VCF01",
    "vcfMangementComponentsInfrastructureSpec": {
        "localRegionNetwork": {
            "networkName": "sfo-m01-cl01-vds01-pg-vm-mgmt",
            "subnetMask": "255.255.255.0",
            "gateway": "10.11.10.1"
        },
        "xRegionNetwork": {
            "networkName": "xint-m01-seg01",
            "subnetMask": "255.255.255.0",
            "gateway": "192.168.11.1"
        }
    }
}

Validate your JSON payload using the POST /v1/vcf-management-components/validations API.

Executing this will return a task id. Copy this id to monitor the task

Check the status of the validation task using GET /v1/vcf-management-components/validations/{validationId} until it’s resultStatus is SUCCEEDED.

Now, submit the same JSON payload to POST /v1/vcf-management-components, and go grab a coffee!

Once the deployment completes, you should have a VCF Operations instance to manage your fleet, along with a VCF Automation instance for the consumption layer.

Upgrading VCF 5.2 to 9.0 – Part 1 – Depot Configuration

The first step in preparing for a VCF 5.2 to 9.0 upgrade is to update your Depot configuration to use token based authentication instead of username & password. This was a change that was made recently to how customers authenticate to the VMware/Broadcom software depots.

The easiest way to do this is to run a Powershell script that is attached to the following KB https://knowledge.broadcom.com/external/article/389276

I am not going to repeat all the pre-reqs and steps from the KB here. The following steps assume you have retrieved your support entitlement token along with taking the necessary backups.

To run the script in interactive mode (with a menu) just run the following

.\VMwareDepotChange.ps1

Run option 1 to set the deployment type and enter Y when prompted to confirm it is a VCF deployment

Enter the SDDC Manager details when prompted and hit enter to validate the credentials

Run option 2 to input your Broadcom download token

Run option 4 to Update your depot configuration to use the token

Once the depot URLs are updated with your token you should now be able to download the required SDDC Manager bundles.

Upgrading VCF 5.2 to 9.0 – Part 2 – Upgrade SDDC Manager

Upgrading VCF 5.2 to 9.0 – Part 3 – Update SDDC Manager Depot

Upgrading VCF 5.2 to 9.0 – Part 4 – Upgrade NSX

Upgrading VCF 5.2 to 9.0 – Part 5 – Upgrade vCenter

Upgrading VCF 5.2 to 9.0 – Part 6 – Transition a vSphere Cluster from vSphere Lifecycle Manager Baselines to Images

Add VMKernels for vSphere Replication using PowerCLI

When configuring vSphere Replication between 2 vCenter Servers, you need to add a dedicated VMKernel to each host in the source & target vCEnter Server clusters. Depending on the number of hosts per cluster, this can be a time-consuming manual task. Here is a quick script leveraging PowerCLI to retrieve the hosst from a specified cluster and loop through them adding a dedicated vSphere Replication VMKernel.

#Source VC vmks
$vCenterServer = "sfo-m01-vc01.sfo.rainpole.io"
$vCenterServerUser = "administrator@vsphere.local"
$vCenterServerPassword = "VMw@re1!VMw@re1!"
$clusterName = "sfo-m01-cl01"
$PortGroupName = "sfo-m01-cl01-vds01-pg-vlr"
$VLANID = 1116
$VSwitch = "sfo-m01-cl01-vds01"
$VMKIP = "10.11.16."  # last octet will be incremented
$lastOctetStart = 101
$SubnetMask = "255.255.255.0"
$mtu = 9000

# Connect to vCenter
Connect-VIServer -Server $vCenterServer -user $vCenterServerUser -password $vCenterServerPassword

# Get Esxi hosts in cluster
$ESXiHosts = Get-cluster -name $clusterName | Get-VMHost

# Loop through each host and add an adapter with vSphere Replication Services enabled
$index = $lastOctetStart
foreach ($ESXi in $ESXiHosts) {
    Write-Host "Processing host: $ESXi"

# Define VMkernel IP for this host
    $VMKIPAddress = "$VMKIP$index"
    $index++

# Add VMkernel adapter
    Write-Host "Adding VMkernel adapter $VMKIPAddress to $ESXi"
    $vmk = New-VMHostNetworkAdapter -VMHost $ESXi -VirtualSwitch $VSwitch -PortGroup $PortGroupName -IP $VMKIPAddress -SubnetMask $SubnetMask -VMotionEnabled $false -mtu $mtu
    $vnicMgr = Get-View -Id $ESXi.ExtensionData.ConfigManager.VirtualNicManager
    $vnicMgr.SelectVnicForNicType('vSphereReplication',$vmk.Name)
$vnicMgr.SelectVnicForNicType('vSphereReplicationNFC',$vmk.Name)

    Write-Host "VMkernel adapter added successfully on $ESXi"
}

# Disconnect from vCenter
Disconnect-VIServer -Confirm:$false

PowerCLI Module For VMware Cloud Foundation: Bringup Using an Existing JSON

This is the 2nd post in a series on the native PowerCLI Module For VMware Cloud Foundation (VCF). If you haven’t seen the previous post, it is available here:

PowerCLI Module For VMware Cloud Foundation: Introduction

This post will focus on the Cloud Builder module to perform a bringup of a VCF instance. For this example, I am using a pre-populated JSON file. I will do a follow-up post on how to create the spec from scratch.

To get started we need a Cloud Builder connection.

Connect-VcfCloudBuilderServer -Server sfo-cb01.sfo.rainpole.io -User admin -Password VMw@re1!VMw@re1!

If you have a pre-populated json spec, you can simply do the following to perform a validation using the Cloud Builder API

$sddcSpec = (Get-Content -Raw .\sfo-m01-bringup-spec.json)
Invoke-VcfCbValidateBringupSpec -SddcSpec $sddcSpec

And once the validation passes, do the following to start the bringup:

Invoke-VcfCbStartBringup -sddcSpec $sddcSpec

Bringup is a long-running task but you can monitor the status using something like this

# Retrieve the bringup task id
$bringupTaskId = (Invoke-VcfCbGetBringupTasks).elements.Id

#Poll the status of the task until it is no longer in progress
Do {
$bringupTask = Invoke-VcfCbGetBringupTaskByID -id $bringupTaskId
}
Until ($bringupTask.Status -ne 'IN_PROGRESS')

QuickTip: Renew SDDC Manager VMCA Certificate

I got a question from someone internally if renewing the VMCA signed certificate on SDDC Manager in a VCF instance is possible. For context, out-of-the-box SDDC Manager is signed by the VMCA on the management domain vCenter Server, but there is no supported way to renew that certificate. So before the VMCA certificate expires, you must replace it with a signed CA cert from your internal CA, or from an external 3rd party CA.

That said, it is possible to leverage VMCA to renew the cert on SDDC Manager. Here are some notes I had from doing this previously in the lab.

Disclaimer: This is not officially supported by VMware/Broadcom, use at your own risk.

First generate a CSR for SDDC Manager in the normal way using the SDDC Manager UI

Download the CSR as sfo-vcf01.sfo.rainpole.io.csr

SSH to the Management vCenter Server and do the following

mkdir /tmp/certs
upload CSR to /tmp/certs
cd /tmp/certs
vi /tmp/certs/cert.cfg

# cert.cfg contents replacing FQDN appropriately
[ req ]
req_extensions = v3_req

[ v3_req ]
extendedKeyUsage = serverAuth, clientAuth
authorityKeyIdentifier=keyid,issuer
authorityInfoAccess = caIssuers;URI:https://sfo-m01-vc01.sfo.rainpole.io/afd/vecs/ca

Save /tmp/certs/cert.cfg

On the management vCenter Server, generate the cert

openssl x509 -req -days 365 -in sfo-vcf01.sfo.rainpole.io.csr -out sfo-vcf01.sfo.rainpole.io.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile cert.cfg

Create a certificate chain

cat sfo-vcf01.sfo.rainpole.io.crt>>sfo-vcf01.sfo.rainpole.io.chain.pem
cat /var/lib/vmware/vmca/root.cer>>sfo-vcf01.sfo.rainpole.io.chain.pem

SSH to SDDC Manager to install the cert

su
cp /etc/ssl/private/vcf_https.key /etc/ssl/private/old_vcf_https.key
mv /var/opt/vmware/vcf/commonsvcs/workdir/vcf_https.key /etc/ssl/private/vcf_https.key
cp /etc/ssl/certs/vcf_https.crt /etc/ssl/certs/old_vcf_https.crt
rm /etc/ssl/certs/vcf_https.crt

SCP sfo-vcf01.sfo.rainpole.io.chain.pem to /etc/ssl/certs/

mv /etc/ssl/certs/sfo-vcf01.sfo.rainpole.io.chain.pem /etc/ssl/certs/vcf_https.crt
chmod 644 /etc/ssl/certs/vcf_https.crt
chmod 640 /etc/ssl/private/vcf_https.key
nginx -t && systemctl reload nginx

You should now have renewed your VMCA signed certificate on SDDC Manager.

PowerCLI Module For VMware Cloud Foundation: Introduction

As you are no doubt aware I am a fan of PowerShell and PowerCLI. Since my early days working with VMware products, whether it was vCenter, vCloud Director or VMware Cloud Foundation (VCF), I have always leveraged PowerCLI to get the job done. Up until recently, there was no native PowerCLI support for the VMware Cloud Foundation API. Hence why I started the open-source PowerVCF project almost 5 years ago! PowerVCF has grown and matured as new maintainers came onboard. Open-source projects are a great way to deliver functionality to our customers that is not yet available in officially supported channels. Since the release of PowerCLI 13.1 I am delighted to say that we now have officially supported, native PowerCLI modules for VMware Cloud Foundation.

2 distinct modules are now part of PowerCLI. One for the Cloud Builder API and one for the SDDC Manager API.

Install-Module -Name VMware.Sdk.Vcf.CloudBuilder
Install-Module -Name VMware.Sdk.Vcf.SddcManager

The cmdlets for each module are too many to list here but to see what’s available once you have them installed do the following

get-command -module VMware.Sdk.Vcf.CloudBuilder
get-command -module VMware.Sdk.Vcf.SDDCManager

You will see from the output that the cmdlets are broken into primarily 2 types:

Initialize-Vcf<xyz>
- Used to gather information and generate input specs
Invoke-Vcf<xyz>
- Used to execute the API request with an input spec

Each module also has a connect/disconnect cmdlet which can be used in the following way

Connect-VcfCloudBuilderServer -Server sfo-cb01.sfo.rainpole.io -User admin -Password VMw@re1!VMw@re1!

This connection object is then stored in $defaultCloudBuilderConnections

Connect-VcfSddcManagerServer -Server sfo-vcf01.sfo.rainpole.io -User administrator@vsphere.local -Password VMw@re1!VMw@re1!

This connection object is then stored in $defaultsddcManagerConnections

Note: If you are working in a lab environment with untrusted certs you can pass -IgnoreInvalidCertificate to each of the above commands.

Once you have an active connection, you can begin to query the API. The example below returns a list of all hosts from SDDC Manager. One thing you will notice, if you are a PowerVCF user, is that you will need to parse the response a little more than you needed to with the PowerVCF cmdlet Get-VCFHost.

Running Invoke-VcfGetHosts will return a list of host elements

So to parse the response, you can do something like this, which will return the details of all hosts

But lets say you would like to filter the response to just the hosts from a specific workload domain. You first need the Id of the workload domain, in this case sfo-m01.

And you can then get a filtered list of hosts for that domain

Hopefully, this introduction was helpful, I will put together a series of blogs over the next few weeks covering some of the main VCF operations, such as bringup, commissioning hosts, deploying workload domains etc. As always, comments & feedback are welcome. Please let me know what your experience is with the new modules and I can feed it back to the engineering team.

Cleanup Failed Credential Tasks in VMware Cloud Foundation

I have covered how to clean up general failed tasks in Cleanup Failed Credentials Tasks in VMware Cloud Foundation in a previous post. Another type of task that can be in a failed state is a credentials rotation operation. Credential operations can fail for a number of reasons (the underlying component is unreachable at the time of the operation etc), and this type of failed task is a blocking task – i.e. you cannot perform another credential task until you clean up or cancel the failed task. The script below leverages the PowerVCF cmdlet Get-VCFCredentialTask to discover failed credential tasks and Stop-VCFCredentialTask to clean them up. As with all scripts, please test thoroughly in a lab before using it in production.

# Script to cleanup failed credential tasks in SDDC Manager
# Written by Brian O'Connell - Staff II Solutions Architect @ VMware
#User Variables
# SDDC Manager FQDN. This is the target that is queried for failed tasks
$sddcManagerFQDN = "sfo-vcf01.sfo.rainpole.io"
# SDDC Manager API User. This is the user that is used to query for failed tasks. Must have the SDDC Manager ADMIN role
$sddcManagerAPIUser = "administrator@vsphere.local"
$sddcManagerAPIPassword = "VMw@re1!"
# DO NOT CHANGE ANYTHING BELOW THIS LINE
#########################################
# Set TLS to 1.2 to avoid certificate mismatch errors
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Install PowerVCF if not already installed
if (!(Get-InstalledModule -name PowerVCF -MinimumVersion 2.4.0 -ErrorAction SilentlyContinue)) {
Install-Module -Name PowerVCF -MinimumVersion 2.4.0 -Force
}
# Request a VCF Token using PowerVCF
Request-VCFToken -fqdn $sddcManagerFQDN -username $sddcManagerAPIUser -password $sddcManagerAPIPassword
# Retrieve a list of failed tasks
$failedTaskIDs = @()
$ids = (Get-VCFCredentialTask -status "Failed").id
Foreach ($id in $ids) {
$failedTaskIDs += ,$id
}
# Cleanup the failed tasks
Foreach ($taskID in $failedTaskIDs) {
Stop-VCFCredentialTask -id $taskID
# Verify the task was deleted
Try {
$verifyTaskDeleted = (Get-VCFCredentialTask -id $taskID)
if (!$verifyTaskDeleted) {
Write-Output "Task ID $taskID Deleted Successfully"
}
}
catch {
Write-Error "Something went wrong. Please check your SDDC Manager state"
}
}