Upgrade vCenter Server Appliance to 6.0 Update 3e

2018, May 27    

Upgrade vCenter Server Appliance to 6.0 Update 3e for spectre compliance (7977753) and disable TLS 1.0.

Table of Contents

Synopsis

If you are like most companies in the world you have been trying to remediate your configuration to ensure it is PCI and Spectre compliant. Spectre-1 (CVE-2017-5753) and Spectre-2 (CVE-2017-5715) both impact vSphere while Meltdown (CVE-2017-5754) is not impacting on these services. For the duration of this write-up I will wrap all of these up into “spectre compliance” and vSphere covers both vCenter services and ESXi. PCI compliance also dictates that TLS 1.0 is disabled by the start of Q3 2018, this means that no service within your vSphere environment should be listening and accepting TLS 1.0 requests. VMware has made disabling TLS 1.0 a fairly painless process, however it requires an updated version of vCenter in order to get the tool to run properly. The process to ensure compliance is a multi-prong approach within the environment I manage due to the amount of VMware services installed. Anytime you are performing upgrades to your vSphere services I highly recommend utilizing the VMware Product Interoperability Matrices in order to ensure all of your products are able to work with one another.

Within my environment I have to perform the upgrades in a very specific order, VMware has released article 2109760 that helps on the specific order that needs to take place. For my configuration I started with vCloud in the article vCloud 5.5.6 to 8.20 Upgrade Process and with this article I have moved onto the vCenter upgrade.

Environment Assumptions

  • This article assumes: The vSphere environment is running in four different
    • Virtual Center vApp (VC)
    • vCloud running in a Linux VM (VCD)
    • NSX vApp (NSX)
    • VMware Update Manager on Windows (VUM)
  • Most of the work will be done from the VMware Update Manager VM.
  • The environment is designed to be as secure as possible; this includes limiting connections into the VMs to ensure compliance.
  • DNS is in place and working as expected prior to upgrading.
  • Chrome is the browser being used in the VUM VM. Other browsers are fine but might require additional/other updates.
  • Multiple Networks are being used in the environment, one for External connections and one for VMware communications and the networks themselves are flat.

Testing & Upgrade Overview

  • Create clones of existing infrastructure.
  • Isolate new VMs in a new network that is not accessible.
  • Configure VMs to communicate with one another.
  • Deploy vCenter Update.
  • Deploy NSX Update.
  • Confirm all services are still working as expected.

TESTING - Cloning and Isolating VMs for upgrade test (Recommended but not required)

NOTE: The latest version of flash needs to be installed on your browser on the Windows VUM VM before proceeding. The update can be obtained for your browser at [adobe]

Perform the following actions within the Virtual Center:

  1. Create a resource pool called “Spectre-Test-Group” within the management gear.
  2. Create two new test network portgroups on a single hypervisor, all can be in an existing vSwitch or a new vSwitch can be created. Since all traffic will remain local to the single hypervisor no actual network changes will have to take place.
    • Spectre-Test-External-VLAN123
    • Spectre-Test-VMware-VLAN456
  3. Clone the 4 VMs: Virtual Center (VC), vCloud (VCD), VUM, and NSX VMs and put them into the new resource pool created, thin provisioned. Example naming convention:
    • test-spectre-VC
    • test-spectre-VCD
    • test-spectre-VUM
    • test-spectre-NSX
  4. Change the networks for all clones to newly created port groups and tag them w/the correct VLAN IDs.
    • External-Network –> Spectre-Test-External-VLAN123
    • VMware-Network –> Spectre-Test-VMware-VLAN456
  5. Create snapshots of all clones. This is not a requirement but makes it easier if any of the following steps fail.
  6. Power up your VMs in the following order: VUM, VC, NSX, and VCD.
  7. All work should be performed within the console session for each VM, do NOT perform this work via SSH/RDP. This is critical because the test VMs are now in a 100% isolated environment, if you SSH/RDP in then you will be working on your actual source VMs.
  8. Since you are now in a network isolated environment we need to install DNS on one of your VMs. Windows DNS Feature can be installed on the VUM or bind can be installed on the VCD VM. Once done, configure a forward lookup zone that matches your existing environment for all devices. Example zone: url.com
    • vc.url.com –> 192.168.1.100
    • vcd.url.com –> 192.168.1.101
    • vum.url.com –> 192.168.1.102
    • NSX.url.com –> 192.168.1.103

      NOTE: It is recommended to utilize the same names and IPs as before, this makes the testing easier, and the IPs should be for whichever network is currently performing the communication between your VMware VMs. IE - If your communication is going across VMware network then DNS will be configured using those IPs.

  9. Edit settings of each VM’s guest OS to point to the new DNS server’s IP address. NOTE: Name resolution is vital for the upgrade process.
    • VC vApp VM - At console press F2 and edit the network settings to update DNS server.
    • VCD Linux VM - VM login and change /etc/resolve.conf with the new DNS server. Your location for resolve.conf may vary depending on what OS you are running.
    • VUM - On both NICs change the first DNS looking to the new IP of itself.
    • NSX - Log into NSX console http://IP.Address click on ‘Settings & Reports’ and then ‘Configuration’ and update DNS.

      NOTE: Depending on your flavor of Linux you might have to log into VCD Linux VM and delete /etc/udev/rules.d/70-persistent-net.rules and confirm MAC addresses are not set in /etc/sysconfig/network-scripts/ifconfig-eth# and reboot. Once reboot is complete ensure the MAC addresses are populated in proper order compared to the NIC ordering. If you needed to change the order, then you will need to reboot again.

  10. From the VUM VM, ensure each device’s web interface to is working properly. EX:
    • https://vc.url.com
    • https://vcd.url.com
    • https://NSX.url.com
  11. Log into VC, VCD, and NSX VMs confirm you have name resolution as well by pinging the FQDN.
    • ping vc.url.com
    • ping vcd.url.com
    • ping NSX.url.com

MAINTENANCE PROCEDURE: Perform vCenter Upgrade to Update 3e.

  1. Create clones on dedicated virtual center vApp.
  2. Create a snapshot on dedicated virtual center vApp and NSX VM, I recommend calling it “vCenter-upgrade-to-6.0u3e”.
  3. Create a baseline to compare your end-results with.
    1. Confirm vCenter client is working as expected.
    2. Confirm vCenter web client is working as expected.
    3. Confirm vCloud UI is working as expected (if applicable).
    4. Ensure a single VM can be built through vCloud (if applicable).
  4. Upgrade vCenter to 6.0 U3e

    NOTE: Confirm freespace in the VCSApp. “df -h” should be run and root should have more than 84% free and /log should have several GB free as well. If you don’t have enough free then clean up log files, sometimes a reboot will help in this area. Click here for more information.

    1. Shutdown vCloud interface gracefully.

      1. Check the status of the vCloud interface, ensure there is nothing going on with it.

         /opt/vmware/vcloud-director/bin/cell-management-tool -u administrator -p PassWord cell -status

      2. To quiesce the cell, and to prevent new jobs from running.

         /opt/vmware/vcloud-director/bin/cell-management-tool -u administrator -p PassWord cell -quiesce true

      3. Recheck the cell status, when the Job Count is 0 then shutdown the cluster.

         /opt/vmware/vcloud-director/bin/cell-management-tool -u administrator -p PassWord cell -status
 /opt/vmware/vcloud-director/bin/cell-management-tool -u administrator -p PassWord cell -shutdown
    2. Download patch “VMware-vCenter-Server-Appliance-6.0.0.30500-7977753-patch-FP.iso” from https://my.vmware.com/group/vmware/patch#search
    3. Patching through GUI (Recommended Method). (Either perform this or through CLI, not both)
      1. Mount the file you downloaded from VMware, you can either do this through the ‘settings’ on the VM or open the console and mount through the CD feature.
      2. Log into https://ip_address_of_vcenter_vapp:5480 using root and the password set for the mgmt GUI.
      3. In Navigator click on “update”
      4. Click on “Check Updates”, and select the CDRom.
      5. Allow the installer to proceed and click “OK” once it has completed. This should take roughly 10 to 15 minutes.
      6. Reboot the vApp by clicking on “Summary” and clicking on “Reboot”

        NOTE: If you want to reboot faster then log into the vApp and run “shutdown -rfn 000” depending on the version. This will skip many of the checks and is only recommended if time is critical.

    4. Patch through via command line (CLI). (Either perform this or through GUI, not both)
      1. Mount the file you downloaded from VMware, you can either do this through the ‘settings’ on the VM or open the console and mount through the CD feature.
      2. At console of the vApp log into the first level ‘command’, do not log fully into shell or the following commands will not work.

      3. To stage ISO:

         software-packages stage --iso
      4. You will have to press enter several times to read the EULA, at the bottom you will have to type “yes” if you go past this then you will have to re-run the “software-packages stage –iso” command agian.

      5. To see the staged content:

         software-packages list --staged

      6. To install the staged patch:

         software-packages install --staged
      7. Reboot the vApp VM by running “shutdown -rfn 000” or “shutdown reboot -r install_update3e” depending on the version.
  5. Install the “VMware vSphere TLS Configurator”
    1. Download “VMware vSphere TLS Configurator” rpm from VMware’s 6.0 page or you can download the mountable iso. Official documentation from VMware
    2. Shutdown vCloud interface gracefully.
    3. Upload file to /tmp or mount it to the vCenter vApp.

    4. Make a temporary directory to extra or mount the files to.

       mkdir /tmp/cdrom

    5. If using the CDRom, then mount the CD rom into the temp directory we created.

       mount /dev/sr0 /tmp/cdrom

    6. Change directory to where the .rpm is located at.

       cd /tmp or /tmp/cdrom

    7. Install the .rpm file.

       rpm -Uvh VMware-vSphereTlsReconfigurator-6.0.0-5051284.x86_64.rpm

    8. Now lets use the tool to scan your configuration to determine current TLS status.

       /usr/lib/vmware-vSphereTlsReconfigurator/VcTlsReconfigurator/reconfigureVc scan

    9. We need to create a backup of your configuration in case there is a problem.

       /usr/lib/vmware-vSphereTlsReconfigurator/VcTlsReconfigurator/reconfigureVc backup

    10. Now lets disable TLS 1.0 in our configuration, leaving just TLSv1.1 and 1.2 enabled:

       /usr/lib/vmware-vSphereTlsReconfigurator/VcTlsReconfigurator/reconfigureVc update -p TLSv1.1 TLSv1.2 --no-restart
    11. Reboot the vApp VM by running “shutdown -rfn 000” or “shutdown reboot -r install_update3e” depending on the version.

      NOTE: VMware says you can simply restart services but I found this only worked most of the time. Rebooting the vApp guaranteed that all services would get restarted and would disable TLS 1.0.

  6. Update NSX package
    1. If your version of NSX is 6.2.2 then you will have to update it as well. 6.2.4 is the first version with a known method to disable TLS 1.0 following this article: https://kb.vmware.com/s/article/2145749
    2. Download the update “NSX for vSphere 6.2.4 Upgrade Bundle” from VMware at https://my.vmware.com/web/vmware/details?productId=417&downloadGroup=NSXV_624
    3. Log into the NSX web interface and click on the “Upgrade” link and click “Upgrade” again.
    4. Provide the location of the downloaded file when prompted.
    5. Wait for the program to upload and analyze the file you provided. This process could take 10 to 15 minutes to complete.
    6. Select “NO” for the two entries and select “Install” to complete. This process could take 10 to 15 minutes to complete. NSX manager will be auto rebooted at the end of this process.
    7. Using Rest API call you can now disable TLS 1.0.
      1. Download and run Advanced REST Client
      2. Do a “GET” command on https://nsxmgr_ip/api/1.0/appliance-management/system/tlssettings, go to authorization tab and enter admin and password of your NSX manager.

      3. Ensure the following is returned.

         <?xml version="1.0" encoding="UTF-8"?>
 <tlsSettings><serverEnabledProtocols>TLS1,TLSv1.1,TLSv1.2</serverEnabledProtocols>
 <clientEnabledProtocols>TLS1,TLSv1.1,TLSv1.2</clientEnabledProtocols>
 </tlsSettings>

      4. Change the method to POST, click on “Body”, change type to “application/xml” and enter the following.

         <?xml version="1.0" encoding="UTF-8"?>
 <tlsSettings><serverEnabledProtocols>TLSv1.1,TLSv1.2</serverEnabledProtocols>
 <clientEnabledProtocols>TLSv1.1,TLSv1.2</clientEnabledProtocols>
 </tlsSettings>
      5. Now click “Send”
      6. Restart NSX server and ensure it properly checks in with vCenter.
    8. Confirm you can see the NSX manager inside vCenter.
    9. Hit rest API for NSX to ensure TLS 1.0 is disabled: https://nsxmgr_ip/api/1.0/appliance-management/system/tlssettings

      NOTE: You might have to log back into NSX manager, click on “NSX Management Service” and re-enter the password information for Lookup and vCenter objects.

  7. Validation

    1. Start the vCloud interface to ensure it is working as expected after the vCenter upgrade:

       service vmware-vcd start
    2. Once all services are back online confirm that vCenter, vCloud, etc are all working as expected. If they are not do not proceed further until you have resolved the issues.
      • vCenter Client Working?
      • vCenter Web Client Working including NSX manager showing in UI?
      • vCloud Web UI working?
    3. Assuming all worked correctly then be sure to delete the snapshots that were created at the beginning of the maintenance.

Rollback Plan (If required)

  • Revert to snapshot created in beginning of maintenance, confirm snapshot allowed the configuration to come back online.
  • If the snapshot restore fails then restore the clone that was created at the beginning of the maintenance, confirm the configuration comes back online.

vSphere Client

If you are still using the original VMware vShere client then be sure you follow my other article to get it working again. vSphere Client and TLS 1.0