Shutdown/Power up a vSAN cluster with PowerCli

I have been doing a lot of lab testing lately and using  vCloud Director is a great way to be able to run side by side tests (sometimes destructive!) against multiple environments without requiring multiple physical clusters. I wanted to emulate a Dell EMC VxRail appliance so I created a 4 node nested vSAN cluster with a vCenter appliance & external PSC running on the cluster, to use as a vCD template. When creating vCD templates it is preferable to power down the environment before adding to the vCD catalog. When it comes to powering down this environment, because the vCenter & PSC are running on the cluster, there is a chicken and egg scenario as you need to power down in this order:

  1. vCenter
  2. PSC
  3. Put ESXi hosts in maintenance mode (vSAN aware operation)
  4. Power down ESXi hosts

The inverse applies when powering up the environment.  So rather than connect to multiple different interfaces to execute the power down or power up operations I put together the following Menu based PowerCli script with the following Options

These options will call the following functions

Option 1. Shutdown vSAN Cluster & PSC/vCenter

  • ConnectViServer $vCenterFQDN $vCenterUser $vCenterPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to vCenter as we need to perform a DRS operation.
  • ChangeDRSLevel $PartiallyAutomated
    • The ChangeDRSLevel function takes an argument for the level to set it to. In this case it sets it to partially Automated to stop DRS from moving VMs around.
  • MoveVMs
    • This function will move the VMs defined in $VMList to the host defined in $VMHost. This ensures that we know where the VMs are when we come to power the cluster back up
  • ConnectViServer $VMHost $VMHostUser $VMHostPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to the host defined in the $VMHost variable
  • ShutdownVM
    • This function will gracefully shutdown the VMs in the order defined in $VMList. In this case it will shutdown the vCenter first & then the PSC. This list can be expanded to include other VMs or could be refactored to use a CSV for a large list of VMs
  • EnterMaintenanceMode
    • This function will put all hosts defined in $VMHosts into maintenance mode. As this is a vSAN cluster it passes the NoAction flag to prevent any data moment or rebuild
  • ShutdownESXiHosts
    • This function will shutdown all hosts in the cluster

Option 2. Startup vSAN Cluster & PSC/vCenter

  • ExitMaintenanceMode
    • This function will exit all hosts from maintenance mode
  • ConnectViServer $VMHost $VMHostUser $VMHostPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to the host defined in the $VMHost variable
  • StartVMs
    • This function will startup the VMs in the reverse order defined in $VMList. In this case it will startup the PSC first & then the vCenter.
  • PollvCenter
    • This function will Poll vCenter until it is up and available
  • ConnectViServer $vCenterFQDN $vCenterUser $vCenterPassword
    • Generic function to connect to a VIServer (ESXi or vCenter). Just pass the hostname, user & password. This instance connects to vCenter as we need to perform a DRS operation
  • ChangeDRSLevel $FullyAutomated
    • The ChangeDRSLevel function takes an argument for the level to set it to. In this case it sets it to Fully Automated as we are done with the maintenance.

The script is posted to Github here and is also posted below. This was written and tested against vSphere 6. I will update it in the coming weeks for vSphere 6.5 and hopefully leverage some of the new APIs

# Script to shutdown & startup a vSAN cluster when vCenter/PSC are running on the cluster
# Created by Brian O'Connell | Dell EMC
# Provided with zero warranty! Please test before using in anger!
# @LifeOfBrianOC
# https://lifeofbrianoc.com/

## User Variables ##
$vCenterFQDN = "vcs01.domain.local"
$vCenterUser = "VC_Admin@domain.local"
$vCenterPassword = "Password123!"
$Cluster = "MARVIN-Virtual-SAN-Cluster"
$VMList = @("VCS01", "PSC01")
$VMHosts = @("esxi04.domain.local", "esxi05.domain.local", "esxi06.domain.local", "esxi07.domain.local")
$VMHost = "esxi04.domain.local"
$VMHostUser = "root"
$VMHostPassword = "Password123!"

### DO NOT MODIFY ANYTHING BELOW THIS LINE ###

# Add Required PowerCli Modules
Get-Module -ListAvailable VM* | Import-Module

# Function to Connect to VI Host (vCenter or ESXi). Pass host, username & password to the function
Function ConnectVIServer ($VIHost, $User, $Password) {
    Write-Host " "
    Write-Host "Connecting to $VIHost..." -Foregroundcolor yellow
    Connect-VIServer $VIHost -User $User -Password $Password | Out-Null
	Write-Host "Connected to $VIHost..." -Foregroundcolor Green
    Write-Host "------------------------------------------------" -Foregroundcolor Green
}

# Define DRS Levels to stop Vms from moving away from the defined host
$PartiallyAutomated = "PartiallyAutomated"
$FullyAutomated = "FullyAutomated"

# Function to Change DRS Automation level						
Function ChangeDRSLevel ($Level) {						

    Write-Host " "
    Write-Host "Changing cluster DRS Automation Level to Partially Automated" -Foregroundcolor yellow
    Get-Cluster $cluster | Set-Cluster -DrsAutomation $Level -confirm:$false | Out-Null
    Write-Host "------------------------------------------------" -Foregroundcolor yellow
}
						
# Function to Move the Vms to a defined host so they can be easily found when starting back up
Function MoveVMs {

    Foreach ($VM in $VMList) {
        # Power down VM
        Write-Host " "
        Write-Host "Moving $VM to $VMHost" -Foregroundcolor yellow
        Get-VM $VM | Move-VM -Destination $VMHost -Confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
    }   
    Disconnect-VIServer $vCenterFQDN -confirm:$false | Out-Null
}

# Function to Shutdown VMs
Function ShutdownVM  {

    Foreach ($VM in $VMList) {
        # Power down VM
        Write-Host " "
        Write-Host "Shutting down $VM" -Foregroundcolor yellow
        Shutdown-VMGuest $VM -Confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "Waiting for $VM to be Shutdown" -Foregroundcolor yellow
        # Check VM powerstate and wait until it is powered off before proceeding with the next VM
        do {
            sleep 15
            $powerState = (get-vm $VM).PowerState
        }
        while ($powerState -eq "PoweredOn")
        Write-Host " "
        Write-Host "$VM Shutdown.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
}

# Function to put all ESXi hosts into maintenance mode with the No Action flag for vSAN data rebuilds
Function EnterMaintenanceMode {

    Foreach ($VMHost in $VMHosts) {
        Connect-VIServer $VMHost -User root -Password $VMHostPassword | Out-Null
        # Put Host into Maintenance Mode
        Write-Host " "
        Write-Host "Putting $VMHost into Maintenance Mode" -Foregroundcolor yellow
        Get-View -ViewType HostSystem -Filter @{"Name" = $VMHost }|?{!$_.Runtime.InMaintenanceMode}|%{$_.EnterMaintenanceMode(0, $false, (new-object VMware.Vim.HostMaintenanceSpec -Property @{vsanMode=(new-object VMware.Vim.VsanHostDecommissionMode -Property @{objectAction=[VMware.Vim.VsanHostDecommissionModeObjectAction]::NoAction})}))}
        Disconnect-VIServer $VMHost -confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "$VMHost in maintenance mode.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
}

# Function to Exit hosts from maintenance mode
Function ExitMaintenanceMode {

    Foreach ($VMHost in $VMHosts) {
        Connect-VIServer $VMHost -User root -Password $VMHostPassword | Out-Null
        # Exit Maintenance Mode
        Write-Host " "
        Write-Host "Exiting Maintenance Mode for $VMHost" -Foregroundcolor yellow
        Set-VMHost $VMHost -State "Connected" | Out-Null
        Disconnect-VIServer $VMHost -confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "$VMHost out of maintenance mode.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
    Write-Host "Waiting for vSAN Cluster to be Online" -Foregroundcolor yellow	
	Sleep 60							
}

# Function to shutdown hosts
Function ShutdownESXiHosts {

    Foreach ($VMHost in $VMHosts) {
        # Exit Maintenance Mode
        Write-Host " "
        Write-Host "Shutting down ESXi Hosts" -Foregroundcolor yellow
        Connect-VIServer -Server $VMHost -User root -Password $VMHostPassword | %{
            Get-VMHost -Server $_ | %{
                $_.ExtensionData.ShutdownHost_Task($TRUE) | Out-Null
            }
        }
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "ESXi host $VMHost shutdown.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }
		Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "All ESXi Hosts shutdown.." -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
}

# Function to Start VMs in the reverse order they were powered down									
Function StartVMs {
    # Reverse the VM list to start in reverse order
    [array]::Reverse($VMList)

    Foreach ($VM in $VMList) {
        # Power on VM
        Write-Host " "
        Write-Host "Powering on $VM" -Foregroundcolor yellow
        Start-VM $VM -Confirm:$false | Out-Null
        Write-Host "------------------------------------------------" -Foregroundcolor yellow
        Write-Host " "
        Write-Host "Waiting for $VM to be Powered On" -Foregroundcolor yellow
        # Check VM powerstate and wait until it is powered on before proceeding with the next VM
        do {
            sleep 15
            $powerState = (get-vm $VM).PowerState
        }
        while ($VM -eq "PoweredOff")
        Write-Host " "
        Write-Host "$VM Powered On..proceeding with next VM" -Foregroundcolor green
        Write-Host "------------------------------------------------" -Foregroundcolor yellow	
    }

}

# Function to Poll the status of vCenter after starting up the VM
Function PollvCenter {

    do 
    {
        try 
        {
            Write-Host " "
            Write-Host "Polling vCenter $vCenterFQDN Availability...." -ForegroundColor Yellow
            Write-Host "------------------------------------------------" -Foregroundcolor yellow
            # Create Web Request
            [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
            $HTTP_Request = [System.Net.WebRequest]::Create("https://$($vCenterFQDN):9443")

            # Get a response
            $HTTP_Response = $HTTP_Request.GetResponse()

            # Get the HTTP code
            $HTTP_Status = [int]$HTTP_Response.StatusCode

            If ($HTTP_Status -eq 200) { 
                Write-Host " "
                Write-Host "vCenter $vCenterFQDN is Available!"  -ForegroundColor Green
                Write-Host "------------------------------------------------" -Foregroundcolor Green
                # Close HTTP request
                $HTTP_Response.Close()
            }
        }
        catch { 
            Write-Host " "
            Write-Host "vCenter $vCenterFQDN Not Available Yet...Retrying Poll..."  -ForegroundColor Cyan
            Write-Host "------------------------------------------------" -Foregroundcolor Cyan
    } }
    While ($HTTP_Status -ne 200)	
}

# Function to display the main menu 
Function Menu 
{
    Clear-Host         
    Do
    {
        Clear-Host                                                                        
        Write-Host -Object 'Please choose an option'
        Write-Host     -Object '**********************'	
        Write-Host -Object 'vCenter & vSAN Maintenance Options' -Foregroundcolor Yellow
        Write-Host     -Object '**********************'
        Write-Host -Object '1.  Shutdown vSAN Cluster & PSC/vCenter '
        Write-Host -Object ''
        Write-Host -Object '2.  Startup vSAN Cluster & PSC/vCenter '
        Write-Host -Object ''
        Write-Host -Object 'Q.  Exit'
        Write-Host -Object $errout
        $Menu = Read-Host -Prompt '(Enter 1 - 2 or Q to quit)'

        switch ($Menu) 
        {
            1 
            {
				ConnectVIServer $vCenterFQDN $vCenterUser $vCenterPassword
                ChangeDRSLevel $PartiallyAutomated
                MoveVMs
                ConnectVIServer $VMHost $VMHostUser $VMHostPassword
                ShutdownVM
                EnterMaintenanceMode
                ShutdownESXiHosts   
            }
            2 
            { 
				ExitMaintenanceMode
                ConnectVIServer $VMHost $VMHostUser $VMHostPassword
                StartVMs
				PollvCenter
                ConnectVIServer $vCenterFQDN $vCenterUser $vCenterPassword
                ChangeDRSLevel $FullyAutomated
            }
            Q 
            {
                Exit
            }	
            default 
            {
                $errout = 'Invalid option please try again........Try 1-2 or Q only'
            }

        }
    }
    until ($Menu -eq 'q')
}   

# Launch The Menu
Menu

Snapshot Management with PowerCli

I’ve been doing some lab work recently, testing out some new automation for deploying Dell EMC Enterprise Hybrid Cloud (EHC) and as part of the testing we needed to be able to quickly and easily create on the fly rollback points at incremental steps in the build to assist with troubleshooting (outside of the standard rollback points created by the EHC Automated Install Tool (AIT) ).

Note: AIT is currently an internal Dell EMC tool used by our professional services organisation to deploy and configure EHC on a customer site.

Disclaimer: As with all scripts and code that you find on the web you should thoroughly test this in a lab environment before considering to use it in production. This script comes with zero warranty as it was something that was created quickly for lab use only!

Snapshots were enough to give us a rollback point, so to achieve this I put together a menu based PowerCli script that will take snapshots of a defined list of VMs with a defined snapshot name, rollback to the last snap taken, or rollback to a defined snapshot.

For more information on how to add a menu to a PowerShell script go here, and for how to add a “Press any key to continue..” to a script go here.

Here is the script for snapshot management. It is broken up into the following PowerShell Functions:

  • CreateVMSnapshot
  • RevertLastVMSnapshot
  • RevertSpecificVMSnapshot
  • anyKey
  • Menu

Before running the script you need to edit the user variables for your environment. The $VMList variable is a comma separated list of VM names as they appear in vCenter. In my example these are the components of a distributed vRA deployment. The $SnapshotName vaiable will be used when creating snapshots or when executing the Revert To Specific Snapshot option

# User Variables
$vCenterFQDN = "vcs01.domain.local"
$vCenterUser = "administrator@vsphere.local"
$vCenterPassword = "Password123!"
$VMList = @("vra01", "vra02", "web01", "web02", "mgr01", "mgr02", "dem01", "dem02", "agt01", "agt02")
$SnapshotName = "Snap01"

To execute the script just browse to the directory you saved the script in a PowerShell or PowerCli console, run ./SnapshotManagement.ps1 and you will be presented with a menu

Select the desired option from the menu. The operations are running Async so are quite quick to complete.

The raw code is below. I’ve also posted the script to GitHub here

# User Variables
$vCenterFQDN = "vcs01.domain.local"
$vCenterUser = "administrator@vsphere.local"
$vCenterPassword = "Password123!"
$VMList = @("vra01", "vra02", "web01", "web02", "mgr01", "mgr02", "dem01", "dem02", "agt01", "agt02")
$SnapshotName = "Snap01"
###############################
# DO NOT EDIT BELOW THIS LINE #
###############################
# Add Required Snappins
Get-Module -ListAvailable VM* | Import-Module

Function CreateVMSnapshot {
# Connect to vCenter
Connect-VIServer $vCenterFQDN -username $vCenterUser -password $vCenterPassword
	Foreach ($VM in $VMList) {
	Write-Host "Creating Snapshot for $VM"
	New-Snapshot -VM $VM -Memory -quiesce -Name $SnapshotName -RunAsync
							 }							 
									}
									
Function RevertLastVMSnapshot {
# Connect to vCenter
Connect-VIServer $vCenterFQDN -username $vCenterUser -password $vCenterPassword
	Foreach ($VM in $VMList) {
	Write-Host "Reverting Snapshot for $VM"
	$snap = Get-Snapshot -VM $VM | Sort-Object -Property Created -Descending | Select -First 1
    Set-VM -VM $vm -SnapShot $snap -Confirm:$false  -RunAsync | Out-Null
							 }							 
									}

Function RevertSpecificVMSnapshot {
# Connect to vCenter
Connect-VIServer $vCenterFQDN -username $vCenterUser -password $vCenterPassword
	Foreach ($VM in $VMList) {
	Write-Host "Reverting Snapshot for $VM"
	#$snap = Get-Snapshot -VM $VM | Sort-Object -Property Created -Descending | Select -First 1
    Set-VM -VM $vm -SnapShot $SnapshotName -Confirm:$false  -RunAsync | Out-Null
							 }							 
									}									

Function anyKey 
{
    Write-Host -NoNewline -Object 'Press any key to return to the main menu...' -ForegroundColor Yellow
    $null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')
    Menu
}
									
Function Menu 
{
    Clear-Host         
    Do
    {
        Clear-Host                                                                        
        Write-Host -Object 'Please choose an option'
        Write-Host     -Object '**********************'	
        Write-Host -Object 'Snapshot VM Options' -ForegroundColor Yellow
        Write-Host     -Object '**********************'
        Write-Host -Object '1.  Snapshot VMs '
		Write-Host -Object ''
        Write-Host -Object '2.  Revert to Last Snapshot '
		Write-Host -Object ''
		Write-Host -Object '3.  Revert To Specific Snapshot '
		Write-Host -Object ''
        Write-Host -Object '4.  Exit'
        Write-Host -Object $errout
        $Menu = Read-Host -Prompt '(0-3)'

        switch ($Menu) 
        {
           1 
            {
                CreateVMSnapshot 			
                anyKey
            }
            2 
            {
                RevertLastVMSnapshot
                anyKey
            }
			3 
            {
                RevertSpecificVMSnapshot
                anyKey
            }
            4 
            {
                Exit
			}	
            default 
            {
                $errout = 'Invalid option please try again........Try 0-4 only'
            }

        }
    }
    until ($Menu -ne '')
}   

# Launch The Menu
Menu

Add “Press any key to continue..” to a PowerShell script

From time to time it is nice to have a “Press any key to continue..” break point in a script to allow the user to review the status of an operation or just to add a user interaction to acknowledge the completion of an operation. This is especially useful when using a menu based script (see here) where the script will revert back to the menu once an operation is complete making it difficult to see the status of an operation when it completes or any Write-Host messages that may have been displayed. To get around this I use the following PowerShell Function to insert a “Press any key to continue..” break point that will wait for the user to…you guessed it…press the any key! 🙂

I use then when using a PowerShell Menu (See more about that here). You can edit the text in the quotes on line 3 to suite your use case. In my case i am calling the Menu function on line 5 so that when a user presses a key it will revert to the script menu. Simples!


Function anyKey
{
Write-Host -NoNewline -Object 'Press any key to return to the main menu...' -ForegroundColor Yellow
$null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')
Menu
}


Adding a simple Menu to a PowerShell script

When sharing scripts that can do multiple operations with other teams I find that adding a simple operator menu that presents a list of choices makes it easier for folks to use the scripts. Here is the simple menu that i add to a lot of my scripts and a quick explanation of how it works.

The menu is created using a PowerShell Function called Menu (This can be anything really but i like to keep it simple!). Within the function we start with a Clear-Host – this will ensure that the menu starts on a clean PowerShell session screen.

In between lines 6 and 17 below you can edit any of the text in the quotes to change the menus item or heading text

Within each menu item you need to callout the PowerShell functions that you want to run as part of the menu option. You can list them one per line as in my example below or on one line with a ; between them (e.g. CreateVMSnapshot; anyKey)

To have the menu launch when the PowerShell script is executed just put the name of the menu function at the bottom of the script outside of any function, see line 53

For more information on the anyKey function go here

EDIT: Modified the exit option to be Q for Quit to keep all menus consistent. Thanks to SiliconBrian for the suggestion!


Function Menu 
{
    Clear-Host         
    Do
    {
        Clear-Host                                                                        
        Write-Host -Object 'Please choose an option'
        Write-Host     -Object '**********************'	
        Write-Host -Object 'Snapshot VM Options' -ForegroundColor Yellow
        Write-Host     -Object '**********************'
        Write-Host -Object '1.  Snapshot VMs '
	Write-Host -Object ''
        Write-Host -Object '2.  Revert to Last Snapshot '
	Write-Host -Object ''
	Write-Host -Object '3.  Revert To Specific Snapshot '
	Write-Host -Object ''
        Write-Host -Object 'Q.  Quit'
        Write-Host -Object $errout
        $Menu = Read-Host -Prompt '(0-3 or Q to Quit)'

        switch ($Menu) 
        {
           1 
            {
                CreateVMSnapshot 			
                anyKey
            }
            2 
            {
                RevertLastVMSnapshot
                anyKey
            }
			3 
            {
                RevertSpecificVMSnapshot
                anyKey
            }
            Q 
            {
                Exit
			}	
            default 
            {
                $errout = 'Invalid option please try again........Try 0-3 or Q only'
            }

        }
    }
    until ($Menu -eq 'q')
}   

# Launch The Menu
Menu

Enable LDAP Auth for RP4VMs

To avoid using local admin accounts it is recommended to integrate RecoverPoint for Virtual Machines (RP4VMs) with AD/LDAP. RP4VMs supports Role Based Access Control (RBAC) where you can create custom roles and assign those custom roles to an AD user or group. To do this you first need to setup LDAP integration. For simplicity this post uses default passwords for the local RP4VM users. Passwords for all local RP users should be changed once the system has been deployed for security reasons.

  1. Connect over SSH to a vRPA in the cluster
  2. Enter username security-admin
  3. Enter the password for the above user (default password is security-admin)
  4. To configure LDAP integration enter config_ldap
  5. To enable Active Directory Support select option 1 and press Enter
  6. Select either LDAP or LDAPs
  7. Enter the primary LDAP server (in my example i use the AD domain as this will allow round robin across all DCs)
  8. Enter the primary LDAP server port
  9. Enter the secondary LDAP server (in my example i left this blank)
  10. Enter the secondary LDAP server port
  11. Enter the base distinguished Name
  12. Enter the search base distinguished name
  13. Enter the bind distinguished name
  14. Enter the password for above AD bind account
  15. Enter a search time limit in seconds. Default is 30. Increase this in larger AD environments

Here is a screenshot of the entire sequence

Once LDAP is configured you then need to create a custom role to assign to an AD user/group.

  1. Connect over SSH to a vRPA in the cluster
  2. Enter username security-admin
  3. Enter the password for the above user (default password is security-admin)
  4. To configure a custom role enter add_role
  5. Enter a name for the new role (e.g. RP4VM_Admins)
  6. Select the desired permissions from the available role permissions. Separate selections with a space
  7. Press Enter when done

Here is a screenshot of the process

Once the role is created you can then assign it to your AD user/group

  1. Connect over SSH to a vRPA in the cluster
  2. Enter username security-admin
  3. Enter the password for the above user (default password is security-admin)
  4. To add a user or group enter add_user (the same function is used to add users & groups)
  5. Select option to add either local user, LDAP user or LDAP group
  6. Enter the username/groupname (sAMAccountName format e.g. the group i am using in this example is called RP4VM_Admins)
  7. Enter the role created previously (e.g. RP4VM_Admins)
  8. For enter group names i tried entering the group name in multiple formats, none of which were accepted so i just hit enter without an entry and it completed
    1. The RP4VM documentation is not clear on what the Group Name field is for but I tested to ensure only users from the desired group have access to the vRPA.

Here is a screenshot of the process

Error restoring SRM placeholder VM

I’ve been doing some lab work this week staging a vSphere 6.0U1b with SRM 6.0 environment for some upgrade scenario testing and i hit an issue with SRM 6.0 that i had not seen before. When trying to restore the SRM placeholder VM for a protected VM I was getting the following error

No hosts with hardware version ‘7’ and datastore(s) “NFS02” which are powered on and not in maintenance mode are available

.srm-placeholder

Seemed like a pretty odd error given that my target host is 6.0 and it has the NFS02 datastore mounted. I checked all the obvious to ensure there were no host issues and then went on the KB hunt. Tried the solution outlined here to no avail https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2079084

Also tried this, again no joy. http://pubs.vmware.com/srm-55/index.jsp?topic=%2Fcom.vmware.srm.admin.doc%2FGUID-FE6A85EC-B44E-415A-9C5F-1E17BC846119.html

As a last ditch effort i tried rebooting the target ESXi host and that fixed the issue and I was then able to restore the placeholder VM and continue testing. Not sure on the root cause. This is a fully nested environment, using vSphere Replication & a VNX File appliance so it may just be environmental. Will update this post if i figure it out!

Onwards with testing!