Upgrading VCF 5.2 to 9.0 – Part 3 – Update SDDC Manager Depot

Once you have upgraded SDDC Manager to 9.0, you will notice the Binary Management screen looks a little different. To enable the depot to download the remaining bundles you need, you must update the download token (again!). To do this, navigate to Depot Settings and select your depot type. In this post I will use the Online Depot. I may create another variant with an offline depot in future.

Under Online Depot, click Edit. Paste your Broadcom support download token and click Authenticate.

You shound then see, Depot Connection Active.

Next, navigate to Binary Management > Upgrade Binaries and select the binaries you need to deploy. In my case I am selecting everything except SDDC Manager.

As I do not have VCF Operations or VCF Automation in my VCF 5.2 environment, and I need to install them as part of VCF 9.0, navigate to Install Binaries and select each one and click Download.

Wait for the downloads to complete before moving on to the next steps.

Upgrading VCF 5.2 to 9.0 – Part 4 – Upgrade NSX
Upgrading VCF 5.2 to 9.0 – Part 5 – Upgrade vCenter
Upgrading VCF 5.2 to 9.0 – Part 6 – Transition a vSphere Cluster from vSphere Lifecycle Manager Baselines to Images

Upgrading VCF 5.2 to 9.0 – Part 2 – Upgrade SDDC Manager

Once you have download the required SDDC Manager bundles, navigate to Workload Domains > Management Workload Domain > Updates, and click Run Precheck and ensure all prechecks pass.

Under Target Version, select VMware Cloud Foundation 9.0.0.0, and click Next.

On the Change Summary screen, click Confirm, and then click Done.

To start the upgrade, click Update Now.

The upgrade of SDDC Manager will begin

Once it completes your SDDC Manager has been upgraded to 9.0

Upgrading VCF 5.2 to 9.0 – Part 3 – Update SDDC Manager Depot
Upgrading VCF 5.2 to 9.0 – Part 4 – Upgrade NSX
Upgrading VCF 5.2 to 9.0 – Part 5 – Upgrade vCenter
Upgrading VCF 5.2 to 9.0 – Part 6 – Transition a vSphere Cluster from vSphere Lifecycle Manager Baselines to Images

Upgrading VCF 5.2 to 9.0 – Part 1 – Depot Configuration

The first step in preparing for a VCF 5.2 to 9.0 upgrade is to update your Depot configuration to use token based authentication instead of username & password. This was a change that was made recently to how customers authenticate to the VMware/Broadcom software depots.

The easiest way to do this is to run a Powershell script that is attached to the following KB https://knowledge.broadcom.com/external/article/389276

I am not going to repeat all the pre-reqs and steps from the KB here. The following steps assume you have retrieved your support entitlement token along with taking the necessary backups.

To run the script in interactive mode (with a menu) just run the following

.\VMwareDepotChange.ps1

Run option 1 to set the deployment type and enter Y when prompted to confirm it is a VCF deployment

Enter the SDDC Manager details when prompted and hit enter to validate the credentials

Run option 2 to input your Broadcom download token

Run option 4 to Update your depot configuration to use the token

Once the depot URLs are updated with your token you should now be able to download the required SDDC Manager bundles.

Upgrading VCF 5.2 to 9.0 – Part 2 – Upgrade SDDC Manager
Upgrading VCF 5.2 to 9.0 – Part 3 – Update SDDC Manager Depot
Upgrading VCF 5.2 to 9.0 – Part 4 – Upgrade NSX
Upgrading VCF 5.2 to 9.0 – Part 5 – Upgrade vCenter
Upgrading VCF 5.2 to 9.0 – Part 6 – Transition a vSphere Cluster from vSphere Lifecycle Manager Baselines to Images

Quick Tip: How to Retrieve VCF 9.0 Autogenerated Passwords From VCF Installer

VCF 9.0 is deployed using the new VCF Installer (replacement for the old Cloud Builder appliance). During a VCF 9.0 bringup, you have the option to let the installer autogenerate all passwords to cut down on inputs. At the end of the installer workflow, there is a button you can click to show all passwords so you can make note of them.

But how do you retrieve them if you closed the installer wizard?

The solution is pretty simple, just navigate to https://vcf-installer-fqdn/vcf-installer-ui/portal/getting-started and under Deploy, click View Details

On the successful deployment screen, click Review Passwords

This will display all passwords used by the VCF Installer for your bringup

Quick Tip: No products found in Aria Lifecycle Manager with VCF 5.2.1

VCF 5.2.1 ships with Aria Lifecycle Manager 8.18. When you attempt to deploy an environment you will be met with the following error:

No content found corresponding to SDDC Manager version 5.2.1 This could be due to version incompatibility between VMware Aria Suite Lifecycle and SDDC Manager.

The reason for this is you need a product support pack (pspak) for Aria LCM 8.18 – specifically VMware Aria Suite Lifecycle 8.18.0 Product Support Pack 3. See this KB for more details on which product support pack maps to which release.

Download the pack from the Broadcom support site and log into Aria LCM. Navigate to Lifecycle Operations > Settings > Product Support Pack and click Upload.

Take a snapshot of Aria LCM and then click Select file and select the product support pack, and click Import.

Monitor the upload process in the Requests pane. Once the upload completes, navigate back to the Product Support Pack screen. The support pack will be shown. Click Apply Version & Submit. Aria LCM will restart services during the install.

Once the install completes, you should not have a list of available products when creating an environment.

Add VMKernels for vSphere Replication using PowerCLI

When configuring vSphere Replication between 2 vCenter Servers, you need to add a dedicated VMKernel to each host in the source & target vCEnter Server clusters. Depending on the number of hosts per cluster, this can be a time-consuming manual task. Here is a quick script leveraging PowerCLI to retrieve the hosst from a specified cluster and loop through them adding a dedicated vSphere Replication VMKernel.

#Source VC vmks
$vCenterServer = "sfo-m01-vc01.sfo.rainpole.io"
$vCenterServerUser = "administrator@vsphere.local"
$vCenterServerPassword = "VMw@re1!VMw@re1!"
$clusterName = "sfo-m01-cl01"
$PortGroupName = "sfo-m01-cl01-vds01-pg-vlr"
$VLANID = 1116
$VSwitch = "sfo-m01-cl01-vds01"
$VMKIP = "10.11.16."  # last octet will be incremented
$lastOctetStart = 101
$SubnetMask = "255.255.255.0"
$mtu = 9000

# Connect to vCenter
Connect-VIServer -Server $vCenterServer -user $vCenterServerUser -password $vCenterServerPassword

# Get Esxi hosts in cluster
$ESXiHosts = Get-cluster -name $clusterName | Get-VMHost

# Loop through each host and add an adapter with vSphere Replication Services enabled
$index = $lastOctetStart
foreach ($ESXi in $ESXiHosts) {
    Write-Host "Processing host: $ESXi"

# Define VMkernel IP for this host
    $VMKIPAddress = "$VMKIP$index"
    $index++

# Add VMkernel adapter
    Write-Host "Adding VMkernel adapter $VMKIPAddress to $ESXi"
    $vmk = New-VMHostNetworkAdapter -VMHost $ESXi -VirtualSwitch $VSwitch -PortGroup $PortGroupName -IP $VMKIPAddress -SubnetMask $SubnetMask -VMotionEnabled $false -mtu $mtu
    $vnicMgr = Get-View -Id $ESXi.ExtensionData.ConfigManager.VirtualNicManager
    $vnicMgr.SelectVnicForNicType('vSphereReplication',$vmk.Name)
$vnicMgr.SelectVnicForNicType('vSphereReplicationNFC',$vmk.Name)

    Write-Host "VMkernel adapter added successfully on $ESXi"
}

# Disconnect from vCenter
Disconnect-VIServer -Confirm:$false

Retrieve VCF Operations Appliance Root Password from the VMware Aria Suite Lifecycle Locker

When you deploy a component using VMware Aria Suite Lifecycle, it stores the credentials in it’s locker. If you need to SSH to a VCF Operations appliance and you dont know the root password, you need to retrieve the root password from the VMware Aria Suite Lifecycle locker. To do this you need to query the Aria Suite Lifecycle API for a list of locker entries using basic auth.

GET https://flt-fm01.rainpole.io/lcm/locker/api/v2/passwords?from=0&size=10

From the response, locate the corresponding vmid for the VCF OPs appliance

{            
"vmid": "a789765f-6cfc-497a-8273-9d8bff2684a5",            "tenant": "default",            
"alias": "VCF-flt-ops01a.rainpole.io-rootUserPassword",          "password": "PASSWORD****",            
"createdOn": 1737740091124,            
"lastUpdatedOn": 1737740091124,            
"referenced": true        
}

Query the Aria Suite Lifecycle locker for the decrypted password, again with basic auth, passing the Aria Suite Lifecycle root password in the payload body.

#BODY (Aria Suite Lifecycle root password)
{
  "rootPassword": "VMw@re1!VMw@re1!"
}

POST https://flt-fm01.rainpole.io/lcm/locker/api/v2/passwords/a789765f-6cfc-497a-8273-9d8bff2684a5/decrypted

If all goes well, it should return the password 

{
    "passwordVmid": "a789765f-6cfc-497a-8273-9d8bff2684a5",
    "password": "u!B1U9#Q5L^o2Vqer@6f"
}

Setup HAProxy as a Load Balancer for VMware VCF Operations

HAProxy is a free opensource load balancer that is supported for use with VMware VCF Operations (formerly VMware Aria Operations). Here are the steps you need to install and configure it on a Debian VM. This configuration was for a lab deployment and may not be optimised for production. Use at your own risk!

EDIT: My colleague Ryan Johnson has very kindly written a shell script to perform the steps. Code available here.

Add an interface to the Debian VM for the VCF Operations VIP


vi /etc/network/interfaces

# Insert the following and save the changes substituting the VLANs/Subnets with your own

# VCF OPs VIP
auto eth1.1110
iface eth1.1110 inet static
address 10.11.10.30
netmask 255.255.255.0
mtu 9000

# Restart network service
systemctl restart networking.service

Install HAProxy

apt-get install haproxy 

# Backup default haproxy.cfg cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

Create a new haproxy.cfg file with the following contents

# Configuration file to load balance VCF Operations
#global parameters 
global
		log 127.0.0.1 local2
		chroot /var/lib/haproxy
		pidfile /var/run/haproxy.pid
		maxconn 2000
		user haproxy
		group haproxy
		daemon
		stats socket /var/lib/haproxy/stats
		ssl-server-verify none
#default parameters unless otherwise specified 
defaults
		log global
		mode http
		option httplog
		option tcplog
		option dontlognull
		timeout connect 5000ms
		timeout client 50000ms
		timeout server 50000ms
#listener settings for stats webpage can be optional but highly recommended listen stats :9090
		balance
		mode http
		stats enable
		stats auth admin:admin
		stats uri /
		stats realm Haproxy\ Statistics
#front settings in this case we bind to all addresses on system or specify an interface
		frontend vrops_frontend_secure
		bind 10.11.10.30:443
		mode tcp
		option tcplog
		default_backend vrops_backend_secure
#backend configuration of receiving servers containing tcp-checks health checks and hashing
		backend vrops_backend_secure
		mode tcp
		option tcplog
		balance source
		hash-type consistent
		option tcp-check
		tcp-check connect port 443 ssl
		tcp-check send GET\ /suite-api/api/deployment/node/status?services=api&services=adminui&services=ui\ HTTP/1.0\r\n\r\n
		tcp-check expect rstring ONLINE
		server node1 10.11.10.31:443 check inter 15s check-ssl maxconn 140 fall 3 rise 3
		server node2 10.11.10.32:443 check inter 15s check-ssl maxconn 140 fall 3 rise 3
		server node3 10.11.10.33:443 check inter 15s check-ssl maxconn 140 fall 3 rise 3

Restart haproxy service

systemctl restart haproxy

You should now be able to browse to https://<aria-operations-vip-fqdn

PowerCLI Module For VMware Cloud Foundation: Bringup Using an Existing JSON

This is the 2nd post in a series on the native PowerCLI Module For VMware Cloud Foundation (VCF). If you haven’t seen the previous post, it is available here:

  1. PowerCLI Module For VMware Cloud Foundation: Introduction

This post will focus on the Cloud Builder module to perform a bringup of a VCF instance. For this example, I am using a pre-populated JSON file. I will do a follow-up post on how to create the spec from scratch.

To get started we need a Cloud Builder connection.

Connect-VcfCloudBuilderServer -Server sfo-cb01.sfo.rainpole.io -User admin -Password VMw@re1!VMw@re1!

If you have a pre-populated json spec, you can simply do the following to perform a validation using the Cloud Builder API

$sddcSpec = (Get-Content -Raw .\sfo-m01-bringup-spec.json)
Invoke-VcfCbValidateBringupSpec -SddcSpec $sddcSpec

And once the validation passes, do the following to start the bringup:

Invoke-VcfCbStartBringup -sddcSpec $sddcSpec

Bringup is a long-running task but you can monitor the status using something like this

# Retrieve the bringup task id
$bringupTaskId = (Invoke-VcfCbGetBringupTasks).elements.Id

#Poll the status of the task until it is no longer in progress
Do {
$bringupTask = Invoke-VcfCbGetBringupTaskByID -id $bringupTaskId
}
Until ($bringupTask.Status -ne 'IN_PROGRESS')