Where Are My VMware Cloud Foundation Logs?

From time to time we all need to look at logs, whether its a failed operation or to trace who did what when. In VMware Cloud Foundation there are many different logs, each one serving a different purpose. Its not always clear which log you should look at for each operation so here is a useful reference table.

Log Type VM Location log Location
BringUp Cloud Builder

JSON Generator – /opt/vmware/sddc-support/cloud_admin_tools/logs/JsonGenerator.log

Platform Audit – /opt/vmware/sddc-support/cloud_admin_tools/logs/PlatformAudit.log

Bringup – /var/log/vmware/vcf/bringup/vcf-bringup-debug.log
Licensing SDDC Manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log
Network Pool SDDC Manager /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
Host Commission/Decommission SDDC Manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log
VI (WLD domain) SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log
vRLI SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log
vROPS SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log
vRA SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log
vRSLCM SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log
Upgrade: /var/log/vmware/vcf/lcm/lcm.log
vRSLCM /var/log/vrslcm/vmware_vrlcm.log
LCM SDDC Manager /var/log/vmware/vcf/lcm/lcm.log
API Login SDDC Manager /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
SoS SDDC Manager /var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log
Certificate Operations SDDC Manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log

vRealize Suite Lifecycle Manager Logs: The Easy Way

vRealize Suite Lifecycle Manager (vRSLCM) is a one stop shop for lifecycle management  (LCM) of your VMware vRealize Suite (vRA, vRB, vROPs, vRLI) . VMware Validate Designs leverages this via Cloud Builder for initial SDDC deployment but it also covers upgrade from a single interface, reducing the need to jump between interfaces by bringing all LCM tasks into a single UI. This doesn’t come without its challenges however, as vRSLCM is now responsible for aggregating all the install/upgrade logs and presenting them in a coherent manner to the user…which isn’t always the case. vRSLCM logs activity in /var/log/vlcm/vrlcm-server.log but at best you get something like this

GET http://localhost:8080/suite/status/1c4a2929-e09c-4a22-b9f1-2834ec1bd65c: 200 null

Which let’s face it isnt very helpful…or is it? At first glance its just a job ID but thanks to @leahy_s in VMware CMBU I can now make this job ID give me more information in a much more structured way, similar to tail -f. Here’s how

And now you should have some readable JSON, hopefully with some more info on the error you are hitting


Managing VMs via the ESXi command line

From time to time a host may be unmanageable from vCenter / web client or you may only have console access. In my case I was bringing up a Dell EMC VxRail. During initial bringup the ESXi hosts do not get a mgmt IP if you do not have DHCP available so management with the web client is not possible. I do have iDRAC access though so can access the console. I needed to see where the VxRail manager VM was running as it comes up during an election process between the hosts. With console access it is still possible to manage VMs using esxcli.

To discover all VMs on a host run the following

  • vim-cmd vmsvc/getallvms

Once you have the output you can use the Vmid to manipulate the powerstate of a VM

  • vim-cmd vmsvc/power.get 2

In my case the VM i wanted was powered off. You can run the following to power it on

  • vim-cmd vmsvc/power.on 2


And there you have it. Simple VM management using vim-cmd. Explore what else you can leverage it for here

Quick Fix: The trust relationship between this workstation and the primary domain failed…

We’ve all been there…attempt to open an RDP session to a VM you haven’t connected to in a while and you see the message above! Traditionally the fix for this was to log on as a local admin user, remove the VM from the AD domain (add to workgroup), reboot, log in again, add to AD domain, reboot….well here is a quicker way of resolving the issue with PowerShell.

Modify the username, password and domain controller FQDN and save the following as ResetDomainMembership.ps1 and run on the affected VM as a local administrator

$password = "Password123!" | ConvertTo-SecureString -asPlainText -Force
$username = "domain\administrator"
$credential = New-Object System.Management.Automation.PSCredential($username,$password)
Reset-ComputerMachinePassword -Server dc01.domain.local -credential $credential
shutdown -r -t 0

Tip: If you dont want to include the password in the script as this is a security concern you can use a Read-Host command to prompt the user for the password

$password = Read-Host -asSecureString "Please enter the password"

Cleanup failed requests in vRA UI

From time to time a request in vRA will fail for whatever reason. When this happens you will see the request status as failed on the requests tab. There is a greyed out delete button that for whatever reason cannot be used to delete the failed request even when logged in as a full tenant/iaas/cloud admin.


There are several reasons you may want to remove failed requests…maybe you may need to deliver a demo to the CIO on some new functionality and failures in the UI never look good…or maybe you just have mild OCD like me and like to cleanup any failures to restore the illusion of all being good with the world! 🙂 Whatever your reasons here is a procedure that you can use.

Disclaimer: I dont believe this procedure if fully supported by VMware so please proceed with caution.

  • SSH to your primary vRA appliance
  • Run the following to view the contents of /etc/vcac/server.xml
    • less /etc/vcac/server.xml
  • Look for the line with password= and copy everything between the “”. This password will allow you to connect to the vRA PostGres DB

  • Run the following command with the password from the above step
    • vcac-config prop-util -d –p “s2enc~K6RsAv5WGpoAt+qsnZPrKErxZ0kU1npeK/G5iMzyaWI=”
  • Next change to the postgres user
    • su postgres
  • Change to the postgres directory
    • cd /opt/vmware/vpostgres/current/bin
  • Connect to the vcac database
    • ./psql vcac -W
  • Enter the password from server.xml
  • vRA requests are store in the cat_request table. To enable us to delete a request we first need the request id. Query the cat_request table for your request ID using the requestnumber (In my case the offending failed requestnumber is 63, as seen in the first column in the screenshot above. replace with your requestnumber)
    • SELECT id,requestnumber FROM cat_request where requestnumber = ’63’;

vRA XaaS blueprint requests are referenced in 1 further table, cat_requestevent. This entriy must be deleted before you can delete the request.

  • Run the following commands to delete the request.
  • delete from cat_requestevent where request_id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;
  • delete from cat_request where id =’4dc74fc2-f855-4eb1-94d6-65481b702acd’;

The offending failed request should now be gone from the requests list in vRA!

Add “Press any key to continue..” to a PowerShell script

From time to time it is nice to have a “Press any key to continue..” break point in a script to allow the user to review the status of an operation or just to add a user interaction to acknowledge the completion of an operation. This is especially useful when using a menu based script (see here) where the script will revert back to the menu once an operation is complete making it difficult to see the status of an operation when it completes or any Write-Host messages that may have been displayed. To get around this I use the following PowerShell Function to insert a “Press any key to continue..” break point that will wait for the user to…you guessed it…press the any key! 🙂

I use then when using a PowerShell Menu (See more about that here). You can edit the text in the quotes on line 3 to suite your use case. In my case i am calling the Menu function on line 5 so that when a user presses a key it will revert to the script menu. Simples!

Function anyKey
Write-Host -NoNewline -Object 'Press any key to return to the main menu...' -ForegroundColor Yellow
$null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')

Error restoring SRM placeholder VM

I’ve been doing some lab work this week staging a vSphere 6.0U1b with SRM 6.0 environment for some upgrade scenario testing and i hit an issue with SRM 6.0 that i had not seen before. When trying to restore the SRM placeholder VM for a protected VM I was getting the following error

No hosts with hardware version ‘7’ and datastore(s) “NFS02” which are powered on and not in maintenance mode are available


Seemed like a pretty odd error given that my target host is 6.0 and it has the NFS02 datastore mounted. I checked all the obvious to ensure there were no host issues and then went on the KB hunt. Tried the solution outlined here to no avail https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2079084

Also tried this, again no joy. http://pubs.vmware.com/srm-55/index.jsp?topic=%2Fcom.vmware.srm.admin.doc%2FGUID-FE6A85EC-B44E-415A-9C5F-1E17BC846119.html

As a last ditch effort i tried rebooting the target ESXi host and that fixed the issue and I was then able to restore the placeholder VM and continue testing. Not sure on the root cause. This is a fully nested environment, using vSphere Replication & a VNX File appliance so it may just be environmental. Will update this post if i figure it out!

Onwards with testing!

Setting up a VMware Appliance update repo

When upgrading VMware appliances using the vami configuration interface the update that is available from VMware does not always match the version you want. mounting an ISO to the VM is not always possible (vCloud Director hosted VMs are an example). Follow this procedure to create a temporary repo to make the correct upgrade bundles available during appliance upgrade
In this example the vRA appliance is used to host the repo. Any VMware appliance can be used for this purpose.
1. SCP the upgrade ISO to /tmp on the vRA appliance
2. SSH to the vRA appliance and navigate to the tmp directory

 cd /tmp

3. Create a new directory called repo

Mkdir repo
 4. Type the following to mount the upgrade ISO to the new repo directory
mount -o loop "iso name" repo
 5. Change directory into repo/update
 cd repo/update
 6. Start the python SimpleHTTPServer
 python -m SimpleHTTPServer
 7. Return to the VAMI configuration interface of the appliance that you are upgrading and browse to the Update tab
 8. Click Settings
 9. Change the Update Repository option to Use Specified Repository
 10. For Repository URL enter http://vRA-FQDN:8000
 Note: Enter the FQDN of the appliance you are using. This may not be the vRA appliance
 11. Click Save Settings
 12. Click Status and click Check Updates
 13. The correct version that you copied to the temporary repo should be available to install

Reload all invalid virtual machines on a single host

From time to time VMs can show in an invalid state in vCenter. It can be difficult to get them back to a normal state. This is a useful command that i found at the end of this KB https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003742

for a in $(vim-cmd vmsvc/getallvms 2>&1 |grep invalid |awk '{print $4}'|cut -d \' -f2);do vim-cmd vmsvc/reload $a;done

If you get an Unexpected token error when running the above command it is probaby due to wordpress messing up the ‘ characters so either replace them or copy the command from the KB!