vCloud 5.5.6 to 8.20 Upgrade Process - Nuthouse.us

vCloud 5.5.6 to 8.20 Upgrade Process and vShield 5.5.4 to NSX 6.2 upgrade process.

Table of Contents:

Synopsis
Environment Assumptions
Testing & Upgrade Overview
TESTING - Cloning and Isolating VMs for upgrade test (Recommended but not required)
PRODUCTION - Cloning and snapshots of VMs
License Keys
Update vShield
Update vCloud
Summary

Synopsis

At the beginning of 2018 a major focus was put upon a vulnerability, called spectre, which was present in virtually all processors built for the last several years. The release of spectre has forced many older systems to be upgraded in order to attempt to bring them into compliance for various standards including Payment Card Industry (PCI). As many system administrators continue to push for updates in their environments it is a delicate balancing act between updates and system availability. In many environments major systems like vCloud Director cannot have much, if any, downtime or there is no administrator on-staff who is technically able to perform the upgrade. Regardless of the reasoning this can leave an administrator with a very old instance of vCloud running in the environment that is not able to support hypervisors, vCenters, or patches that are spectre compliant.

I recently moved into a role that had vCloud 5.5.6 installed along with vShield 5.5.4. After a quick conversation with VMware they confirmed that vCloud 5.5.6 and vShield 5.5.4 were not going to receive any updates for spectre compliance. Therefore best solution is to upgrade the existing installations to a newer version of vCloud and NSX. As an alternative, you can stand up a new instance of vCloud and NSX and migrate your configuration, however that is very time consuming and typically requires migrations of vCloud information. To make the process as simple as possible our department decided upon upgrading vCloud to 8.20 with NSX 6.2. Since this is an in-place upgrade, all information was retained within vCloud and all functionality we utilize continued to work after the upgrade. We also obtained many new features with the new vCloud such as enabling/disabling hot-plug, time sync, etc. from within the vCloud UI/API. This helps our department greatly as we utilize vCloud as a front end UI for several system administrators to self-manage their own environments.

Interoperability Matrix

vCloud Director Upgrade Path
NSX to vCloud Director
ESXi to vCloud Director

NOTE: ESXi 6.0 U3d is not expressly listed at the time this document was written.

The latest ESXi 6.0 that is spectre compliant is not compatible with vCloud 5.5.6. The newest version of vCloud which supports ESXi 6.0 that is spectre compliant is 8.10 but there is no direct upgrade path to vCloud 8.10 from vCloud 5.5.6. However, vCloud 5.5.6 has a direct upgrade path to either 8.02 or 8.20. Since 8.02 is not spectre compliant we have to go with vCloud Director 8.20. In my environment this caused me another issue since vShield Endpoint 5.5.4 is not compatible with vCloud Director 8.20. vCloud Director 8.20 requires minimum NSX 6.20 installed. These are the reasons behind writing a formal process, specific to my environment, for upgrading from vCloud 5.5.6 to 8.20 and vShield 5.5.4 to NSX 6.2.

This article goes into the detailed process of how to test the vCloud upgrade process in a cloned/isolated environment, confirm it works, and then perform the process in the real environment. Following this model will allow testing of the design without causing any downtime and confirm it will work as needed. It also gives two different ways to restore the environment in case of failures during the actual upgrade process.

Note: Any old snapshots that are in place holding onto ‘ghosted’ information /must/ be deleted. ‘Ghosted’ objects are things like old port groups that have been removed since the snapshot has been created, and will typically show up in italics in the vSphere client. Failure to remove these have a high chance of causing the upgrade process to fail during the database upgrade portion. The error can occur in both the testing phase and the real upgrade phase.

Environment Assumptions

This article assumes: The vCloud environment is running in four different VMs: Virtual Center vApp (VC), vCloud running in a Linux VM (VCD), vShield vApp (vShield), and VMware Update Manager on Windows (VUM).
Most of the work will be done from the VMware Update Manager VM.
The environment is designed to be as secure as possible; this includes limiting connections into the VMs to ensure compliance.
DNS is in place and working as expected prior to upgrading.
Chrome is the browser being used in the VUM VM. Other browsers are fine but might require additional/other updates.
Multiple Networks are being used in the environment, one for External connections and one for VMware communications and the networks themselves are flat.
vCloud database is stored on VUM VM inside an MS SQL database.

Testing & Upgrade Overview

Create clones of existing infrastructure.
Isolate new VMs in a new network that is not accessible.
Configure VMs to communicate with one another.
Deploy vShield Update.
Deploy vCloud Update.
Confirm all services are still working as expected.

TESTING - Cloning and Isolating VMs for upgrade test (Recommended but not required)

NOTE: The latest version of flash needs to be installed on your browser on the Windows VUM VM before proceeding. The update can be obtained for your browser at adobe

Perform the following actions within the Virtual Center:

Create a resource pool called “Spectre-Test-Group” within the management gear.
Create two new test network portgroups on a single hypervisor, all can be in an existing vSwitch or a new vSwitch can be created. Since all traffic will remain local to the single hypervisor no actual network changes will have to take place.
- Spectre-Test-External-VLAN123
- Spectre-Test-VMware-VLAN456
Clone the 4 VMs: Virtual Center (VC), vCloud (VCD), VUM, and vShield VMs and put them into the new resource pool created, thin provisioned. Example naming convention:
- test-spectre-VC
- test-spectre-VCD
- test-spectre-VUM
- test-spectre-vShield
Change the networks for all clones to newly created port groups and tag them w/the correct VLAN IDs.
- External-Network –> Spectre-Test-External-VLAN123
- VMware-Network –> Spectre-Test-VMware-VLAN456
Create snapshots of all clones. This is not a requirement but makes it easier if any of the following steps fail.
Power up your VMs in the following order: VUM, VC, vShield, and VCD.
All work should be performed within the console session for each VM, do NOT perform this work via SSH/RDP. This is critical because the test VMs are now in a 100% isolated environment, if you SSH/RDP in then you will be working on your actual source VMs.
Since you are now in a network isolated environment we need to install DNS on one of your VMs. Windows DNS Feature can be installed on the VUM or bind can be installed on the VCD VM. Once done, configure a forward lookup zone that matches your existing environment for all devices. Example zone: url.com

NOTE: It is recommended to utilize the same names and IPs as before, this makes the testing easier, and the IPs should be for whichever network is currently performing the communication between your VMware VMs. IE - If your communication is going across VMware network then DNS will be configured using those IPs.
- vc.url.com –> 192.168.1.100
- vcd.url.com –> 192.168.1.101
- vum.url.com –> 192.168.1.102
- vshield.url.com –> 192.168.1.103
Edit settings of each VM’s guest OS to point to the new DNS server’s IP address. NOTE: Name resolution is vital for the upgrade process.
- VC vApp VM - At console press F2 and edit the network settings to update DNS server.
- VCD Linux VM - VM login and change /etc/resolve.conf with the new DNS server. Your location for resolve.conf may vary depending on what OS you are running.
- VUM - On both NICs change the first DNS looking to the new IP of itself.
- vShield - Log into vShield console http://IP.Address click on ‘Settings & Reports’ and then ‘Configuration’ and update DNS.
  
  NOTE: Depending on your flavor of Linux you might have to log into VCD Linux VM and delete /etc/udev/rules.d/70-persistent-net.rules and confirm MAC addresses are not set in /etc/sysconfig/network-scripts/ifconfig-eth# and reboot. Once reboot is complete ensure the MAC addresses are populated in proper order compared to the NIC ordering. If you needed to change the order, then you will need to reboot again.
From the VUM VM, ensure each device’s web interface to is working properly. EX: https://vc.url.com https://vcd.url.com https://vshield.url.com
Log into VC, VCD, and vShield VMs confirm you have name resolution as well by pinging the FQDN. ping vc.url.com ping vcd.url.com ping vshield.url.com

PRODUCTION - Cloning and snapshots of VMs

NOTE: The latest version of flash needs to be installed on your browser on the Windows VUM VM before proceeding. The update can be obtained for your browser at adobe

Create resource pool called “Spectre-Backup-Group” within the management gear.
Clone VC, VCD, VUM, and vShield VMs and put them into the new resource pool created, thin provisioned. These are for restore purposes in case of a total system failure. Example naming convention:
- Spectre-BU-VC
- Spectre-BU-VCD
- Spectre-BU-VUM
- Spectre-BU-vShield
Create snapshots of all original VMs titled “vCloud-Upgrade”
Create a baseline for testing at the end. Log into the primary vCloud UI, ensure the interface is working as expected. Attempt to build a new vApp and power it on, if it fails then troubleshoot the issue. If you cannot resolve the issue, then do not proceed. If other systems utilize API calls, then test those systems as well.
Log into each device’s web interface to ensure services are working. EX: https://vc.url.com https://vcd.url.com https://vshield.url.com
It is recommended to take screenshots of the screens so you can compare the vCloud and vShield objects after the upgrade.

License Keys

vCloud License Key - vCloud 5.5.6 license key will probably not work on vCloud 8.20. It will be important to note this ahead of time, so you can obtain upgraded license keys or new keys prior to the upgrade. Please consult your VMware account manager for more information. Log into vCloud interface as an administrator. Click on “Administration” Click on “License” Copy Serial Number.
vShield License Key - vShield Endpoint is enabled using your vSphere license key and therefore does not require a separate license.
NSX 6.2 - A new license key will be required for this.

Update vShield

Log into VUM VM.
Download VMware-vShield-Manager-upgrade-bundle-6.2.2-3604087.tar.gz from VMware. “vCNS to NSX for vSphere 6.2.2 Upgrade Bundle”

NOTE 1: If you cannot download from within the VUM then download to your PowerCLI machine and uploading using PowerCLI command “Copy-VMGuestFile”. (Syntax is added as footnote)

NOTE 2: If the file is too large to upload with Copy-VMGuestFile then split the file with 7zip (650MB) and upload 7zip following the same method.
Log into vShield web interface, click on ‘updates’ and ‘upload upgrade bundle’ and then ‘choose file’
Select the saved location for VMware-vShield-Manager-upgrade-bundle-6.2.2-3604087.tar.gz and click ‘upload file’.
Review information and select ‘install’
Select ‘confirm install’ and wait for the process to complete.
When the VM goes down for the reboot power off the VM and adjust RAM to 16GB, a requirement for NSX manager.
Log into the new NSX interface using https://vshield.url.com Username: admin Password: default Click on “Summary” and ensure all services are started; enable SSH as well. Start any that are not running. Navigate to “Manage Application Settings” Click on “NSX Management Service” and ensure both lookup and vCenter services are showing “connected”. If showing disconnected, then log into the console for vShield and confirm it has name resolution to the domain for the services. If still showing disconnected, then re-enter credentials within NSX.

Update NSX password (This step can be skipped on the test environment) Log in to the NSX vApp at console.

 username: admin
 password: default
 manager> enable
 password:
 manager#

Switch to Configuration mode:

 manager# configure terminal

Change the admin account password:

 manager(config)# cli password ENTER-NEW-PASSWORD

Save the configuration:

 manager(config)# write memory
 Building Configuration...
 Configuration saved.
 [OK]

Confirm NSX is working - in the VUM VM navigate to https://vc.url.com/vsphere-client and confirm “Networking & Security” is displayed on the navigator menu.
In the VUM VM navigate to https://vshield0.url.com and confirm you can login and your structure looks the same.

Update vCloud

Backup the vCloud database.
- Log into the VUM VM, launch “SQL Server Management Studio”
- Click “Connect” on the default instance displayed. If none displayed, then connect to the local machine.
- Expand the “database” section.
- Right click on “vCloud” –> “Tasks” –> “Backups”
- In destination click on “Disk” and then “Add”
- You can use the default location and add ‘vcloud.bak’ at the end for the filename and click ok twice.
- Click “OK” once the job is complete and then close SQL Management Studio.
  
  NOTE: A great suggestion from Shane Jameson is to first shutdown vCloud Director services, as listed in the following steps, and then perform the backups. This ensures there are no active processes being performed against vCloud Director during the backup process.
Log into the vCloud VM.
Download vmware-vcloud-director-distribution-8.20.0-5070903.bin from VMware and upload to /opt/vmware/vcloud-director/bin “VMware vCloud Director 8.20”

NOTE 1: If you cannot download from within the VUM then download to your PowerCLI machine and uploading using PowerCLI command “Copy-VMGuestFile”. (Syntax is added as footnote)

NOTE 2: If the file is too large to upload with Copy-VMGuestFile then split the file with 7zip (650MB) and upload 7zip following the same method.
Change directory:
```
 cd /opt/vmware/vcloud-director/bin
```

Change the file to executable:

 chmod u+x vmware-vcloud-director-distribution-8.20.0-5070903.bin

To check the cell status:

 ./cell-management-tool -u administrator -p password cell -status

To quiesce the cell, and to prevent new jobs from running:

 ./cell-management-tool -u administrator -p password cell -quiesce true

Recheck the cell status, when the Job Count is 0 then shutdown the cluster:

 ./cell-management-tool -u administrator -p password cell -status
 ./cell-management-tool -u administrator -p password cell -shutdown

As a general maintenance clear the QRTZ tables. Log into the VUM VM and launch MS SQL Server Management Studio. Confirm the db was backed up previously, if not re-run the backup process.

 Delete from task;
 update jobs set status = 3 where status = 1;
 update last_jobs set status = 3 where status = 1;
 delete from busy_object;
 delete from QRTZ_SCHEDULER_STATE;
 delete from QRTZ_FIRED_TRIGGERS;
 delete from QRTZ_PAUSED_TRIGGER_GRPS;
 delete from QRTZ_CALENDARS;
 delete from QRTZ_TRIGGER_LISTENERS;
 delete from QRTZ_BLOB_TRIGGERS;
 delete from QRTZ_CRON_TRIGGERS;
 delete from QRTZ_SIMPLE_TRIGGERS;
 delete from QRTZ_TRIGGERS;
 delete from QRTZ_JOB_LISTENERS;
 delete from QRTZ_JOB_DETAILS;
 delete from compute_resource_inv;
 delete from custom_field_manager_inv;
 delete from cluster_compute_resource_inv;
 delete from datacenter_inv;
 delete from datacenter_network_inv;
 delete from datastore_inv;
 delete from dv_portgroup_inv;
 delete from dv_switch_inv;
 delete from folder_inv;
 delete from managed_server_inv;
 delete from managed_server_datastore_inv;
 delete from managed_server_network_inv;
 delete from network_inv;
 delete from resource_pool_inv;
 delete from storage_pod_inv;
 delete from task_inv;
 delete from vm_inv;
 delete from property_map;

Execute the script.

In the VCD VM run the installation file with the following command:
```
 ./vmware-vcloud-director-distribution-8.20.0-5070903.bin
```
The installer starts and then prompts you if you wish to continue the upgrade, enter: Y

Change the timeout value against the SQL DB. Back up the global.properties file using this command:

 cp /opt/vmware/vcloud-director/etc/global.properties /opt/vmware/vcloud-director/etc/global.properties.bak

Open the file using a text editor:

 vi /opt/vmware/vcloud-director/etc/global.properties

Locate this entry within the file:

 database.jdbcUrl = jdbc:jtds:sqlserver:\/\/IP.ADDRESS.HERE:1433\/vcloud;socketTimeout=90

Change this entry to:

 database.jdbcUrl = jdbc:jtds:sqlserver:\/\/IP.ADDRESS.HERE:1433\/vcloud;socketTimeout=900

Upgrade the database:
```
 /opt/vmware/vcloud-director/bin/upgrade
```
Enter “Y” to ‘Do you wish to upgrade the product now?’ Enter “Y” for ‘Do you wish to upgrade the database which cells are still active?’ Enter “Y” for ‘Backup is complete.’ Enter “Y” or “N” for ‘join customer experience.’

NOTE: If you have updated vShield to NSX and it fails saying it is not upgraded then you need to manually change DB to reflect new version of NSX. It will fail with the following status:
```
 found one or more unsupported vShield Manager or NSX servers.
 Identifier: BBADBBADBBADBBADBBADBBADBBADBBAD
 VSM/NSX IP: 10.16.27.14
 VSM/NSX Version 5.5
 Associated vCenter Name: vc.url.com
 One or more vShield Manager or NSX servers registered to this vCloud Director installation is not supported by the version of vCloud Director you are upgrading to.  Upgrade canceled, please follow the procedures in the vSM/NSX upgrade guide to upgrade those unsupported vSM/NSX servers to one of these supported vSM/NSX versions: 5.2, 6.3
```
To Edit the DB, launch SQL server management studio, connect to the vCloud db, restore the backup of the database, then open tables and go down to dbo.vshield_manager, right click and ‘edit the top 200 rows’, change the entry for vsm_version from ‘5.5’ to ‘6.2’ and close out the editor.
Answer “Yes” to auto start vCloud again.

If you were using VMRC in vCloud 5.5.6 then you will also need to explicitly change the console session type being used. Create a backup of the MS SQL database ‘vCloud’ again. Right click on vCloud database and click on “new query” and enter the following and then click ‘execute’:

 insert into config (cat, name, value) values ('vcloud', 'console.webmks.firefox.windows', 'false');
 insert into config (cat, name, value) values ('vcloud', 'console.webmks.ie.windows', 'false');
 insert into config (cat, name, value) values ('vcloud', 'console.webmks.chrome.windows', 'false');
 insert into config (cat, name, value) values ('vcloud', 'console.webmks.firefox.linux', 'false');
 insert into config (cat, name, value) VALUES ('vcloud', 'console.webmks.chrome.linux', 'false');

The upgrade is now complete. Test if the UI and any API calls are still working as expected.

Summary I hope you found this article helpful, it was my goal to show both testing methodology and the process for upgrading vCloud and vShield. While the overall process should work in any environment, please be sure to tailor it to fit your environmental variables to ensure success.

Footnote: copy-vmguestfile -source “C:\folder\filename.zip” -Destination C:\folder -VM VUM.url.com -localtoguest -HostUser root -HostPassword root_pass -GuestUser administrator -GuestPassword admin_pass