VCF Recovery Models

Protection & more importantly, recovery of VMware Cloud Foundation (VCF) is something I and Ken Gould have worked closely on for a number of years now. Whether it was a VVD based deployment or in more recent years, a VCF deployment, we have tested, validated & documented the process & procedures you need to follow to be successful, should you need to recover from a failure.

When we talk about VCF Recovery Models, there are 3 main pillars.

Backup & Restore

This is the traditional scenario, where one or more components have failed and you need to recover/restore them in-place. The table below shows what is required for each VCF 9.0 component in the event of a single component failure. https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vvs/9-X/site-protection-and-disaster-recovery-for-vmware-cloud-foundation.html

The VCF 9.0 documentation for component-level backup & restore covers the manual steps required to configure backup for each component, and then how to recover/restore each component, and is available here https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/backup-and-restore-of-cloud-foundation.html

VCF Instance Recovery

VCF Instance Recovery takes things a step further where an entire VCF Instance has failed and needs to be recovered. This could be due to a site outage (where a recovery site is not available), a catastrophic hardware failure, or a cyber attack. Many financial customers also have regulatory requirements like DORA (Digital Operational Resilience Act) where they must be able to demonstrate to the regulator that they can bring their critical infrastructure back in the case of a cyber attack. The table below shows what is required for each VCF 9.0 component in the event of an entire VCF Instance failure.

The VCF 9.0 documentation for VCF Instance Recovery leverages backups for the components that support it, and redeploy for those that don’t to bring the VCF instance back with the same identity as it had before it went down. We also provide a PowerShell module to automate many of the manual, potentially error-prone & onerous tasks to expedite the recovery time. The documentation is available here https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/-vmware-cloud-foundation-instance-recovery.html

VCF Disaster Recovery

VCF Disaster Recovery is used to protect a VCF Instance across physical sites/regions. It requires a second, recovery VCF Instance available in another site/region with IP mobility between the instances. How you provide the IP mobility is up to you. As VMware, we would love you to use NSX Federation, but you can just as easily use fabric enabled stretched L2. For VCF 9.0 we use a combination of VMware Live Recovery & redeploy/restore from backup to recover the management components from the protected to the recovery sites. The same process can be used to recover business workloads.

The table below shows how each VCF 9.0 component is recovered on the recovery site in the event of an entire VCF site failure.

The VCF 9.0 documentation for VCF Disaster Recovery was published as a validated solution. It leverages live recovery replication and recovery plans for the components that support it, and redeploy & restore from backup for those that don’t. The documentation is available here https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vvs/9-X/site-protection-and-disaster-recovery-for-vmware-cloud-foundation.html

In the next major VCF release we are planning to bring these VCF Recovery Model pillars together so they are easier to find and compare side by side, and also add additional ones around ransomware recovery and lateral security to round out the VCF Protection & Recovery story. Credit to my colleague Tom Harrington for the colourful tables!

Upgrading VCF 5.2 to 9.0 – Part 9 – Deactivate Enhanced Link Mode (ELM)

The final part of this series on Upgrading VCF 5.2 to 9.0 is to Deactivate Enhanced Link Mode (ELM) on the VCF instance that we have upgraded. ELM has been around forever, but with VCF 9.0, ELM is now deprecated. You can no longer deploy vCenter instances in an ELM ring. VCF 9.0 introduces a new concept of vCenter linking, which, along with VCF SSO, enabled by VCF Identity Broker (VIDB), replaces the functionality previously provided by ELM. I will cover vCenter linking and VCF SSO in a later post, but before you can take advantage of VCF SSO, you must first deactivate ELM.

Deactivating ELM means each vCenter in the ELM ring now becomes a standalone vCenter with its own isolated SSO domain and its is an all or nothing operation, meaning you cannot selectively remove one vCenter from the ELM ring. Once you perform the operation, all vCenters will be removed from the ELM ring, and all vCenters will use their own instance of vsphere.local.

Important Note: Taking offline snapshots of all vCenter instances in the ELM ring is recommended in case you need to revert.

To deactivate ELM, we will use the SDDC Manager API.

  • Browse to SDDC manager and click on Developer Center
  • Navigate to Domains and under GET /v1/domains click Execute.
  • Expand the response and locate the id for the management domain
  • Expand POST /v1/domains/{id}/validations , enter the management domain id in the id field and the following json in the body, and click Execute.
{
  "breakElmSpec": {
      "isReconcileWorkflow": false
  }
}
  • Expand GET /v1/domains/{id}/validations/{validation_id} and replace {id} with the management domain ID. Replace {validation_id} with the validation ID, from the previous step, and click Execute. Expand the response to ensure it is successful.

To break ELM across all VCF domains in the VCF instance, expand PATCH /v1/domains/{id} , enter the management domain id in the id field and the following json in the body, and click Execute.

{
  "breakElmSpec": {
      "isReconcileWorkflow": false
  }
}

Locate the task ID in the response.

To monitor the task progress, expand GET /v1/tasks/{id} , enter the task id in the id field, and click Execute.

Adding an Existing VCF Instance to a VCF Fleet

This is a scenario that is not covered very well in our current VCF 9.0 docs (I am working to rectify that), where a customer has more than 1 existing VCF 5.x instance and they want to move to VCF 9.0. The upgrade of the first instance and the addition (or upgrade) of VCF Operations & VCF Automation are documented. But how do you go about adding those additional VCF 5.x instances into your new VCF 9.0 fleet? Although it looks like a long post, the process is pretty straightforward.

TL;DR Version

  • Create an integration in VCF Operations for the new VCF instance.
  • Update the integration to use a system-managed service account.
  • Add the new management domain vCenter as a deployment target in fleet management.
  • Deploy a VCF Operations collector to the new instance.
  • Update the integration to use the new collector/collector group.

Step By Step Version

First, you should upgrade the additional instances to 9.0. This is not a mandatory step (and sometimes may not be possible for operational reasons), but it will set you up to be able to consume all of the features of VCF 9.0.

Next, create a new VMware Cloud Foundation integration in VCF Operations for the SDDC Manager of the additional VCF instance.

  • In the VCF Operation UI, navigate to Administration > Integrations, and click Add.
  • Click VMware Cloud Foundation on the Account Types screen.
  • Enter a friendly name for the instance and enter the FQDN for SDDC Manager.
  • To add a new credential, click the + next to credential
  • Enter a username and password with the ADMIN role in SDDC Manager, and click OK. For now, I am using administrator@vsphere.local as we will change this to a service account later.
  • Leave the Collector / Group as Default for now. This will also be changed later.
  • Click Validate Connection and accept the certificate.
  • Expand Advanced Settings, change Enable Domain Monitoring on Creation to true, and remove the VCF Configuration limits File Name (if it is populated).
  • To add the connection, click Add.
  • Acknowledge the Account Monitoring Notification and click OK.
  • On the Integrations screen, expand VMware Cloud Foundation and expand the new Integration. You will see that the workload domain components are in a stopped status. To start them, click the vertical ellipsis at the VCF instance level and click Start Collecting All.

To move away from using a defined credential to a system managed service account, click the vertical ellipsis at the VCF instance level and click Edit.

  • On the Cloud Account Information screen, under System Managed Credential, click Activate, and click Save.
  • You can now safely delete the ADMIN role credential you created earlier. On the Integrations screen, click Credentials.
  • On the Credentials screen, click the vertical ellipsis next to the credential you created and click Delete.
  • Click Yes to confirm the delete operation.

Next you need to add the management domain vCenter as a deployment target to allow you to deploy VCF Operations collector(s).

  • Navigate to Fleet Management > Lifecycle > Settings > Deployment Targets, and click Add Deployment Target.

The Add Deployment Target wizard will find vCenter instances that are configured as a VCF integration and present a list to choose from.

  • Select the vCenter and click Validate.
  • Once the validation is successful, click Add.
  • Monitor the task status until it completes.
  • To deploy VCF Operations collector(s) to the newly added VCF instance, navigate to Components and click operations in the components list.
  • Before adding new nodes to the VCf Operations cluster, it is advisable to perform an inventory sync to avoid any drift. Click Trigger Inventory Sync.
  • On the Trigger Inventory Sync pop-up, click Submit.
  • Monitor the inventory sync task until it completes and then navigate back to the Add Nodes screen and click Add Node. As you have already performed the Inventory Sync, it can be ignored here. Click Proceed.
  • On the Infrastructure screen, select the newly added management domain vCenter from the drop down and select the target cluster, network and datastore and click Next.
  • On the Networks screen, do not make any changes. You will set the network information on the collector directly. Click Next.
  • On the Components screen, uncheck Take component snapshot, as it is not required when adding collector nodes.
  • Before you can deploy the collector, you need to add a root user password for the collector to the credentials store. On the Components screen, click Add Password.
  • Enter the password details and click Add.
  • Scroll down to Components and click the + to add a component. From the list, select Cloud Proxy. Note: Cloud Proxy is the legacy name for the VCF Operations Collector.
  • Enter the details for the collector and click the settings icon on the right.
  • Update the network details to reflect the newly added VCF instance
  • If additional NTP servers are required, under NTP Settings, click Add New Server. Enter the server details and click Submit.
  • To change the NTP Server selection, click Edit Server Selection.
  • On the Select Servers screen, select the correct server(s) and click Next. If required, adjust the server priority and click Finish.
  • Under Root Password, click Select Root Password and select the root password created earlier.
  • Under Timezone, select the correct timezone from the dropdown and click Save.
  • On the Components screen, click Next.
  • On the Precheck screen, click Run Precheck, and wait for the precheck to complete successfully. Once its complete, click Next.
  • On the Summary screen, click Submit to deploy the collector(s).
  • Monitor the deployment task to completion.
  • On the Integrations screen, expand VMware Cloud Foundation and expand the new Integration. Click the vertical ellipsis at the VCF instance level and click Edit.
  • Under Collector / Group, select the newly deployed collector, and click Validate Connection.

That takes care of the instance-level setting, which will be applied to any new workload domains. You must repeat the process at a per-workload domain level for any existing domains.

  • Click the vertical ellipsis at the domain level, and click Edit.
  • On the vCenter screen, change the Collector / group to the new collector and click Validate Connection. Click Ok on the Test Connection popup and click Save.
  • Now you should see your new VCF instance using the correct collector/group.

If you are using VCF Automation, your new VCF instance should now be discovered automatically.

Quick Tip: Retrieve vSphere Supervisor Control Plane SSH Password

SSH access to the Supervisor Control Plane VMs uses an auto-generated password. The password for the system user on these VMs needs to be retrieved from associated vCenter.

IMPORTANT: Direct access to the vSphere Supervisor Control Plane VMs should be used with caution and for troubleshooting purposes only.

SSH to the vCenter as the root user, and enter the bash shell by typing shell

Run the following to retrieve the password

root@sfo-w01-vc01 [ ~ ]# /usr/lib/vmware-wcp/decryptK8Pwd.py

You should now be able to SSH to the IP of the vSphere Supervisor Control Plane VM (The IP returned will be the floating IP or FIP) with username root.

Upgrading VCF 5.2 to 9.0 – Part 8 – Deploy VCF Fleet Management Components

VCF 9.0 introduced the concept of VCF fleet, which is defined as:

An environment that is managed by a single set of fleet-level management components – VCF Operations & VCF Automation. A VCF fleet contains one or more VCF Instances and may contain one or more standalone vCenter instances, managed by the VCF Operations instance for the fleet. The management domain of the first VCF Instance in the VCF fleet typically hosts the fleet-level management components.

When deploying a new VCF fleet, you get the option to deploy the fleet-level management components using the VCF installer. Because I am upgrading from VCF 5.2, where I did not have Aria Operations or Aria Automation, I need to deploy new instances of each component (If I had pre-existing instances, they could be upgraded). You can deploy them manually from OVA, however, there is a new SDDC Manager API to automate the process using a JSON payload.

The API can be accessed via the SDDC Manager developer centre, under VCF Management Components.

The JSON payload to deploy VCF Operations (including a collector & the fleet management appliance) and VCF Automation is as follows: (NOTE: This spec is for a simple/single node deployment of the fleet management components where VCF Operations & VCF Automation will be deployed to an NSX Overlay segment, and the VCF Operations collector will be deployed to the management DVPG)

 {
    "vcfOperationsFleetManagementSpec": {
        "hostname": "flt-fm01.rainpole.io",
        "rootUserPassword": "VMw@re1!VMw@re1!",
        "adminUserPassword": "VMw@re1!VMw@re1!",
        "useExistingDeployment": false
    },
    "vcfOperationsSpec": {
        "nodes": [
            {
                "hostname": "flt-ops01a.rainpole.io",
                "rootUserPassword": "VMw@re1!VMw@re1!",
                "type": "master"
            }
        ],
        "useExistingDeployment": false,
        "applianceSize": "medium",
        "adminUserPassword": "VMw@re1!VMw@re1!"
    },
    "vcfOperationsCollectorSpec": {
        "hostname": "sfo-opsc01.sfo.rainpole.io",
        "rootUserPassword": "VMw@re1!VMw@re1!",
        "applianceSize": "small"
    },
    "vcfAutomationSpec": {
        "hostname": "flt-auto01.rainpole.io",
        "adminUserPassword": "VMw@re1!VMw@re1!",
        "useExistingDeployment": false,
        "ipPool": [
            "192.168.11.51",
            "192.168.11.52"
        ],
        "internalClusterCidr": "250.0.0.0/15",
        "vmNamePrefix": "flt-auto01"
    },
    "vcfInstanceName": "San Francisco VCF01",
    "vcfMangementComponentsInfrastructureSpec": {
        "localRegionNetwork": {
            "networkName": "sfo-m01-cl01-vds01-pg-vm-mgmt",
            "subnetMask": "255.255.255.0",
            "gateway": "10.11.10.1"
        },
        "xRegionNetwork": {
            "networkName": "xint-m01-seg01",
            "subnetMask": "255.255.255.0",
            "gateway": "192.168.11.1"
        }
    }
}

Validate your JSON payload using the POST /v1/vcf-management-components/validations API.

Executing this will return a task id. Copy this id to monitor the task

Check the status of the validation task using GET /v1/vcf-management-components/validations/{validationId} until it’s resultStatus is SUCCEEDED.

Now, submit the same JSON payload to POST /v1/vcf-management-components, and go grab a coffee!

Once the deployment completes, you should have a VCF Operations instance to manage your fleet, along with a VCF Automation instance for the consumption layer.

Upgrading VCF 5.2 to 9.0 – Part 1 – Depot Configuration

The first step in preparing for a VCF 5.2 to 9.0 upgrade is to update your Depot configuration to use token based authentication instead of username & password. This was a change that was made recently to how customers authenticate to the VMware/Broadcom software depots.

The easiest way to do this is to run a Powershell script that is attached to the following KB https://knowledge.broadcom.com/external/article/389276

I am not going to repeat all the pre-reqs and steps from the KB here. The following steps assume you have retrieved your support entitlement token along with taking the necessary backups.

To run the script in interactive mode (with a menu) just run the following

.\VMwareDepotChange.ps1

Run option 1 to set the deployment type and enter Y when prompted to confirm it is a VCF deployment

Enter the SDDC Manager details when prompted and hit enter to validate the credentials

Run option 2 to input your Broadcom download token

Run option 4 to Update your depot configuration to use the token

Once the depot URLs are updated with your token you should now be able to download the required SDDC Manager bundles.

Add VMKernels for vSphere Replication using PowerCLI

When configuring vSphere Replication between 2 vCenter Servers, you need to add a dedicated VMKernel to each host in the source & target vCEnter Server clusters. Depending on the number of hosts per cluster, this can be a time-consuming manual task. Here is a quick script leveraging PowerCLI to retrieve the hosst from a specified cluster and loop through them adding a dedicated vSphere Replication VMKernel.

#Source VC vmks
$vCenterServer = "sfo-m01-vc01.sfo.rainpole.io"
$vCenterServerUser = "administrator@vsphere.local"
$vCenterServerPassword = "VMw@re1!VMw@re1!"
$clusterName = "sfo-m01-cl01"
$PortGroupName = "sfo-m01-cl01-vds01-pg-vlr"
$VLANID = 1116
$VSwitch = "sfo-m01-cl01-vds01"
$VMKIP = "10.11.16."  # last octet will be incremented
$lastOctetStart = 101
$SubnetMask = "255.255.255.0"
$mtu = 9000

# Connect to vCenter
Connect-VIServer -Server $vCenterServer -user $vCenterServerUser -password $vCenterServerPassword

# Get Esxi hosts in cluster
$ESXiHosts = Get-cluster -name $clusterName | Get-VMHost

# Loop through each host and add an adapter with vSphere Replication Services enabled
$index = $lastOctetStart
foreach ($ESXi in $ESXiHosts) {
    Write-Host "Processing host: $ESXi"

# Define VMkernel IP for this host
    $VMKIPAddress = "$VMKIP$index"
    $index++

# Add VMkernel adapter
    Write-Host "Adding VMkernel adapter $VMKIPAddress to $ESXi"
    $vmk = New-VMHostNetworkAdapter -VMHost $ESXi -VirtualSwitch $VSwitch -PortGroup $PortGroupName -IP $VMKIPAddress -SubnetMask $SubnetMask -VMotionEnabled $false -mtu $mtu
    $vnicMgr = Get-View -Id $ESXi.ExtensionData.ConfigManager.VirtualNicManager
    $vnicMgr.SelectVnicForNicType('vSphereReplication',$vmk.Name)
$vnicMgr.SelectVnicForNicType('vSphereReplicationNFC',$vmk.Name)

    Write-Host "VMkernel adapter added successfully on $ESXi"
}

# Disconnect from vCenter
Disconnect-VIServer -Confirm:$false

PowerCLI Module For VMware Cloud Foundation: Bringup Using an Existing JSON

This is the 2nd post in a series on the native PowerCLI Module For VMware Cloud Foundation (VCF). If you haven’t seen the previous post, it is available here:

  1. PowerCLI Module For VMware Cloud Foundation: Introduction

This post will focus on the Cloud Builder module to perform a bringup of a VCF instance. For this example, I am using a pre-populated JSON file. I will do a follow-up post on how to create the spec from scratch.

To get started we need a Cloud Builder connection.

Connect-VcfCloudBuilderServer -Server sfo-cb01.sfo.rainpole.io -User admin -Password VMw@re1!VMw@re1!

If you have a pre-populated json spec, you can simply do the following to perform a validation using the Cloud Builder API

$sddcSpec = (Get-Content -Raw .\sfo-m01-bringup-spec.json)
Invoke-VcfCbValidateBringupSpec -SddcSpec $sddcSpec

And once the validation passes, do the following to start the bringup:

Invoke-VcfCbStartBringup -sddcSpec $sddcSpec

Bringup is a long-running task but you can monitor the status using something like this

# Retrieve the bringup task id
$bringupTaskId = (Invoke-VcfCbGetBringupTasks).elements.Id

#Poll the status of the task until it is no longer in progress
Do {
$bringupTask = Invoke-VcfCbGetBringupTaskByID -id $bringupTaskId
}
Until ($bringupTask.Status -ne 'IN_PROGRESS')

QuickTip: Renew SDDC Manager VMCA Certificate

I got a question from someone internally if renewing the VMCA signed certificate on SDDC Manager in a VCF instance is possible. For context, out-of-the-box SDDC Manager is signed by the VMCA on the management domain vCenter Server, but there is no supported way to renew that certificate. So before the VMCA certificate expires, you must replace it with a signed CA cert from your internal CA, or from an external 3rd party CA.

That said, it is possible to leverage VMCA to renew the cert on SDDC Manager. Here are some notes I had from doing this previously in the lab.

Disclaimer: This is not officially supported by VMware/Broadcom, use at your own risk.

First generate a CSR for SDDC Manager in the normal way using the SDDC Manager UI

Download the CSR as sfo-vcf01.sfo.rainpole.io.csr

SSH to the Management vCenter Server and do the following

    mkdir /tmp/certs
    upload CSR to /tmp/certs
    cd /tmp/certs
    vi /tmp/certs/cert.cfg
    
    # cert.cfg contents replacing FQDN appropriately
    [ req ]
    req_extensions = v3_req
    
    [ v3_req ]
    extendedKeyUsage = serverAuth, clientAuth
    authorityKeyIdentifier=keyid,issuer
    authorityInfoAccess = caIssuers;URI:https://sfo-m01-vc01.sfo.rainpole.io/afd/vecs/ca
    
    Save /tmp/certs/cert.cfg

    On the management vCenter Server, generate the cert

    openssl x509 -req -days 365 -in sfo-vcf01.sfo.rainpole.io.csr -out sfo-vcf01.sfo.rainpole.io.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile cert.cfg
    

    Create a certificate chain

    cat sfo-vcf01.sfo.rainpole.io.crt>>sfo-vcf01.sfo.rainpole.io.chain.pem
    cat /var/lib/vmware/vmca/root.cer>>sfo-vcf01.sfo.rainpole.io.chain.pem
    

    SSH to SDDC Manager to install the cert

    su
    cp /etc/ssl/private/vcf_https.key /etc/ssl/private/old_vcf_https.key
    mv /var/opt/vmware/vcf/commonsvcs/workdir/vcf_https.key /etc/ssl/private/vcf_https.key
    cp /etc/ssl/certs/vcf_https.crt /etc/ssl/certs/old_vcf_https.crt
    rm /etc/ssl/certs/vcf_https.crt
    
    SCP sfo-vcf01.sfo.rainpole.io.chain.pem to /etc/ssl/certs/
    
    mv /etc/ssl/certs/sfo-vcf01.sfo.rainpole.io.chain.pem /etc/ssl/certs/vcf_https.crt
    chmod 644 /etc/ssl/certs/vcf_https.crt
    chmod 640 /etc/ssl/private/vcf_https.key
    nginx -t && systemctl reload nginx

    You should now have renewed your VMCA signed certificate on SDDC Manager.

    PowerCLI Module For VMware Cloud Foundation: Introduction

    As you are no doubt aware I am a fan of PowerShell and PowerCLI. Since my early days working with VMware products, whether it was vCenter, vCloud Director or VMware Cloud Foundation (VCF), I have always leveraged PowerCLI to get the job done. Up until recently, there was no native PowerCLI support for the VMware Cloud Foundation API. Hence why I started the open-source PowerVCF project almost 5 years ago! PowerVCF has grown and matured as new maintainers came onboard. Open-source projects are a great way to deliver functionality to our customers that is not yet available in officially supported channels. Since the release of PowerCLI 13.1 I am delighted to say that we now have officially supported, native PowerCLI modules for VMware Cloud Foundation.

    2 distinct modules are now part of PowerCLI. One for the Cloud Builder API and one for the SDDC Manager API.

    Install-Module -Name VMware.Sdk.Vcf.CloudBuilder
    Install-Module -Name VMware.Sdk.Vcf.SddcManager

    The cmdlets for each module are too many to list here but to see what’s available once you have them installed do the following

    get-command -module VMware.Sdk.Vcf.CloudBuilder
    get-command -module VMware.Sdk.Vcf.SDDCManager

    You will see from the output that the cmdlets are broken into primarily 2 types:

    • Initialize-Vcf<xyz>
      • Used to gather information and generate input specs
    • Invoke-Vcf<xyz>
      • Used to execute the API request with an input spec

    Each module also has a connect/disconnect cmdlet which can be used in the following way

    Connect-VcfCloudBuilderServer -Server sfo-cb01.sfo.rainpole.io -User admin -Password VMw@re1!VMw@re1!

    This connection object is then stored in $defaultCloudBuilderConnections

    Connect-VcfSddcManagerServer -Server sfo-vcf01.sfo.rainpole.io -User administrator@vsphere.local -Password VMw@re1!VMw@re1!

    This connection object is then stored in $defaultsddcManagerConnections

    Note: If you are working in a lab environment with untrusted certs you can pass -IgnoreInvalidCertificate to each of the above commands.

    Once you have an active connection, you can begin to query the API. The example below returns a list of all hosts from SDDC Manager. One thing you will notice, if you are a PowerVCF user, is that you will need to parse the response a little more than you needed to with the PowerVCF cmdlet Get-VCFHost.

    Running Invoke-VcfGetHosts will return a list of host elements

    So to parse the response, you can do something like this, which will return the details of all hosts

    But lets say you would like to filter the response to just the hosts from a specific workload domain. You first need the Id of the workload domain, in this case sfo-m01.

    And you can then get a filtered list of hosts for that domain

    Hopefully, this introduction was helpful, I will put together a series of blogs over the next few weeks covering some of the main VCF operations, such as bringup, commissioning hosts, deploying workload domains etc. As always, comments & feedback are welcome. Please let me know what your experience is with the new modules and I can feed it back to the engineering team.