Seamless Computing

List of what should be backed up

There is some overlap between components listed below – some are within the vCenter database, but should still be exported separately (such as vDS’s). Other components can be disregarded if you have VM-level backups that can perform file-level restores. This is not a complete list, but it might be helpful for some items that are commonly overlooked (like the Inventory Service database).

1. vCenter (KB 1023985 – Backing Up & Restoring vCenter):
   1. vCenter Database
   2. vCenter SSL certificates
      1. Windows Server 2012: <install volume>\ProgramData\VMware\VMware VirtualCenter\SSL
   3. vpxd.cfg
      1. vpxd.cfg stores things like:
         1. If you increased the default timeout value for tasks that get sent through vCenter (KB 1017253)
         2. Created Custom Attributes (migrate these to vCenter Tags if possible)
      2. File path for vpxd.cfg:
         1. Windows Server 2012: <install volume>\ProgramData\VMware\VMware VirtualCenter
      3. Permissions structure
         1. You can export and import the vCenter permissions hierarchy using Powershell functions. See the reference material below:
            1. VMware vSphere PowerCLI Reference: Automating vSphere Administration (Kindle Location 832).
2. vCenter Inventory Service Database
   1. Inventory Service performs searches/index functions within the VI/Web Client
   2. The Inventory Service is a service, but also uses a database. This flat-file database contains:
      1. vCenter tags
      2. Storage Profiles
      3. Storage Capabilities
      4. Index of which VMs have which Storage Profile assigned
   3. Location of Inventory Service database: <install volume>\Program Files\VMware\Infrastructure\Inventory Service\

3. ESXi hosts & VMs
4. DRS configuration
5. Virtual Distributed Switches (KB 2034602)
6. Single Sign On
7. VMware Update Manager
8. Syslog collector
9. Misc add-ons/plugins
10. Configure Syslog to capture at least 24 hours worth of logs
11. Documentation
    1. Location of backed up data and retention info
    2. Non-standard installation paths
    3. Service account usernames and passwords:
       1. 64 bit ODBC for vCenter
       2. 32 bit ODBC for VUM
       3. Account for registering/running VUM
       4. Account for running vCenter services
       5. Account for running SQL services
       6. Account for connecting Web Client to vCenter
       7. Root password(s) for ESXi hosts
       8. Password for “vi-admin” on vMA appliance
       9. Other misc usernames/passwords (Prosphere, Veeam/TSM/Commvault/BE, VSI Plugin, etc)

BACKUP – Inventory Service Database

• You can take a file-level backup of the entire Inventory Service folder (path below), but it’s unclear what affect this will have on vCenter if a restore is performed using this method. The recommended way to get a backup of this is to use the built-in scripts designed to take a backup of the inventory service database. This can be done with Windows Task Scheduler
• Task scheduler configuration settings:
  • Action: Start a program
  • Program/script: backup.bat
  • Add arguments: -file <install volume>\<custom subfolder>\inventorydb-backup
  • Start in: <install volume>\Program Files\VMware\Infrastructure\Inventory Service\scripts\
• Once you verify that the script runs, right click on the task and select Export. This will create an XML file
• After the scheduled task runs, you can have the backup team come by and pick up the “inventorydb-backup” and the exported Task Scheduler XML files

RESTORE – Inventory Service Database

• If you need to roll back to a previous backup of the inventory database, you will lose any changes/additions/deletions to the following:
  • vCenter tags (version 5.1+ only)
  • Storage Profiles
  • Storage Capabilities
  • Index of which VMs have which Storage Profile assigned
• The following steps are taken from KB 2017370:
  • Stop the vCenter Inventory Service and VMware vSphere Profile-Driven Storage Service
  • Open a command prompt and CD to <install volume>\Program Files\VMware\Infrastructure\Inventory Service\scripts
  • Type restore -backup <path to backup file>\<backup filename>
  • Start the vCenter Inventory Service & VMware vSphere Profile-Driven Storage service

BACKUP – ESXi Host

• Source link
• A partial list of what gets backed up:
  • Virtual Standard Switches
  • DNS & Routing
  • Services & Firewall
  • NTP
  • iSCSI Software Adapter
• A partial list of what does NOT get backed up:
  • Datastore names
  • Virtual machine names within the inventory list
• Open PowerCLI and connect to the vCenter server managing the host(s)
• Single Host Backup: Get-VMhostFirmware -VMHost esxi1.qa.local -BackupConfiguration -DestinationPath D:\VMware_DR
• All Hosts Backup: Get-VMHost | Get-VMhostFirmware -BackupConfiguration -DestinationPath D:\VMware_DR
• Once complete, it will create the file “configBundle-esxi1.qa.local.tgz” in the case of a single host backup operation

RESTORE – ESXi Host

• Place the host in maintenance mode
• Open PowerCLI and connect to the vCenter server managing the host
• Set-VMHostFirmware -VMHost esxi1.qa.local -Restore -SourcePath <source path of where you backed up the ESXi host configuration, including the filename>
• Enter root credentials for the ESXi host
• The ESXi host will reboot automatically
• Ideally you would rebuild the host, but if you need to get a host back up and running quickly this is a good option

BACKUP – vCenter Database

• There are multiple ways to backup the vCenter database, depending on what type of database you use. Refer to the specific software vendors documentation for more information

RESTORE – vCenter Database

• Shut down the vCenter server
• Restore a known-good copy of the vCenter database
• Power on the vCenter server
• Note: virtual machine inventory names & vSS port group settings (among others) are not stored within the vCenter database – they are local to the ESXi host, so they will be unaffected by the restore

BACKUP – vCenter Update Manager (VUM)

• There are multiple ways to backup the VUM database, depending on what type of database you use. Refer to the specific software vendors documentation for more information
• Take note of any non-standard settings, in addition to:
  • Baselines / Baseline Groups
  • All settings within the Configuration section (Download settings/schedule, etc)

RESTORE – vCenter Update Manager (VUM)

• Stop the VMware vSphere Update Manager service
• Restore a known-good copy of the VUM database
• Start the VMware vSphere Update Manager service
• Reconfigure VUM using the documented configuration settings

BACKUP – VMware Syslog Collector

• Backup everything within C:\ProgramData\VMware\VMware Syslog Collector on the server that has Syslog Collector installed. The syslog data path does not change regardless whether it is a standalone or vCenter integrated installation

RESTORE – VMware Syslog Collector

• There is no database, so you can restore the missing syslog data from a file-level backup
• The reinstallation is straightforward, so just dump the syslog data back into C:\ProgramData\VMware\VMware Syslog Collector. There may be additional steps if a custom certificate is used

BACKUP/RESTORE – vCenter DRS Rules

• Run the two scripts located here

BACKUP – vCenter SSO

Backing up and restoring the VMware vCenter Single Sign-On 5.5 configuration (2057353)

Prerequisites

• Service account with Administrator access to the vCenter server
• Create a folder entitled “DisasterRecovery” on a data volume on the vCenter server
• Use third-party software to backup the entire “DisasterRecovery” folder sometime after all of the individual vCenter SSO backup tasks have run successfully each day

Generate Log Bundle

• Create a daily scheduled task with these parameters:
  • Program/script: cscript
  • Arguments: sso-support.wsf /s:C:\DisasterRecovery
  • Start in: C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\vm-support
  • Run whether the user is logged in or not
  • Run with highest privileges
• Once you verify that the script runs, right click on the task and select Export

Backup Windows Registry Keys

• Add the DR service account or group to this key with Read access: HKLM\SYSTEM\CurrentControlSet\services\VMwareDirectoryService
• Create a daily scheduled task with these parameters:
  • Program/script: reg
  • Arguments: export HKLM\SYSTEM\CurrentControlSet\services\VMwareDirectoryService c:\DisasterRecovery\SSORegistryBackup.reg /y
  • Start in: C:\Windows\system32
  • Run whether the user is logged in or not
  • Run with highest privileges
• Once you verify that the script runs, right click on the task and select Export

Backup Windows Data

• SSL Certificates:
  • C:\ProgramData\VMware\CIS\runtime\VMwareSTS\conf
• Certificate server data:
  • C:\ProgramData\VMware\CIS\data\vmca
• KDC data:
  • C:\ProgramData\VMware\CIS\cfg\vmkdcd
  • C:\ProgramData\MIT\Kerberos5

Backup VMware Directory Service Database

• Create a sub-folder within the “DisasterRecovery” folder entitled “VMdirBackup”
• Create a daily scheduled task with these parameters:
  • Program/script: vdcbackup
  • Arguments: C:\ProgramData\VMware\cis\data\vmdird C:\DisasterRecovery\VMdirBackup
  • Start in: C:\Program Files\VMware\Infrastructure\VMware\CIS\vmdird
  • Run whether the user is logged in or not
  • Run with highest privileges
• Once you verify that the script runs, right click on the task and select Export

RESTORE – vCenter SSO

Follow the section entitled “Restoring the vCenter SSO 5.5 configuration” (KB 2057353)