This is a modernized version of the "Single use vihostupdate How To for ESXi 4.x, which was a modernized version of the "Single use ESXUPDATE How To for ESX 4"  post I wrote a long time ago. With vSphere 5, vihostupdate and esxupdate are not applicable for ESXi 5. We instead will need to use the esxcli command to apply the updates to our ESXi 5 hosts.

01: Make sure you have the vMA 5.0 or the vCLI installed and configured or that you have ESXi Shell access on the ESXi 5 host.

02: Download the patch bundle directly from VMware Support. This download will be .zip file.  Do not extract it.

03: Upload the .zip file to a datastore that is accessible on the ESXi host you wish to update. The syntax below will use /vmfs/volumes/datastore1, and you may need to adjust as necessary. Note that the .zip file is uploaded to the ESXi host.

Note:  In the examples below, the syntax is specific for the vMA. Adjust accordingly, if you are using another approach.

04: Obtain local console access to the vMA and login with the vi-admin account.

05. To determine if the host needs to be placed in maintenance mode, issue the following command:

esxcli --server=10.10.10.10 --username=root software sources vib get -d /vmfs/volumes/datastore1/ESXi500-201109001.zip | grep "Maintenance Mode Required: True"

06.  If grep returns "Maintenance Mode Required: True" results, then issue the following command to place the host in maintenance mode:

vicfg-hostops --server 10.10.10.10 --operation enter

07. Verify that the host is in maintenance mode, by issuing the following command:

vicfg-hostops --server=10.10.10.10 --operation info

Note: You could also use the vSphere Client to put the ESXi 5 host in maintenance mode.

08. To verify which VIBs are already installed on the ESXi 5 host, issue the following command:

esxcli --server=10.10.10.10 --username=root software vib list | more

09. To find out which VIBs are available in the depot (the downloaded .zip file), issue the following command:

esxcli --server=10.10.10.10 --username=root software sources vib list --depot=/vmfs/volumes/datastore1/ESXi500-201109001.zip | more

10. To update the ESXi 5 host with the VIBs included in the depot, issue the following command:

esxcli --server=10.10.10.10 --username=root software vib update --depot=/vmfs/volumes/datastore1/ESXi500-201109001.zip

11. When the update is complete, verify the information presented. If prompted, reboot the ESXi 5 host by issuing the following command:

vicfg-hostops --server 10.10.10.10 --operation reboot

12. Verify the patch bundle was installed, by issuing the following command:

esxcli --server=10.10.10.10 --username=root software vib list | more

13. If applicable, take the ESXi 5 host out of maintenance mode using the vSphere Client or with the following command:

vicfg-hostops --server 10.10.10.10 --operation exit

As always, thanks for reading!

It has been a long time in the works, but I am happy to announce that I am currently in the process of writing Sybex's VCP5 study guide. The book is tentatively titled "VCP: VMware Certified Professional on vSphere 5." This book will be a complete rewrite with 100% new content to focus on the VCP5. I am also pleased to announce that Troy Clavell has signed on to be technical editor for this project.

The goal of the book is to follow the VCP5 exam blueprint and cover each of the objectives in detail. I also hope to impart real world practical knowledge along the way, using many of the things learned here in the VMTN forums. The book will also include many hands-on exercises, 20 review questions at the end of each chapter, a glossary and plenty of helpful tips. There will also be a companion CD that will include two 75-question bonus exams and at least 100 flashcards.

The book will be available in the first quarter of 2012, but I will announce the actual availability when I know for certain. If you have any feedback on what you think would make a great VCP5 book, I would love to hear it.

Thanks,
Brian

vSphere 4.0 introduced virtual hardware version 7, which added hot plug support for virtual devices and hot add support for memory and virtual CPUs.  Finding the operating systems that could be used with hot add was a somewhat difficult process, and there wasn't a readily available "official" resource that could be used for reference.  With the release of v 2.0 of the VMware Compatibility Guide, finding this information just got a lot easier.  So, what operating systems can be used with hot add memory and vCPU?

Memory

vCPU

There are many!  Rather than posting large tables that would require constant updating, I posted the direct link to the VMware Compatibility Guide.  This way the results shown will always be current.

Thanks for reading.

Brian

This is an updated version of my blog entry "SQL Server Data Consistency Levels with VCB Backups" from more than a year ago, and vSphere 4.1 and the VMware vStorage APIs for Data Protection have changed things up a bit.

The chart below summarizes the current state of data consistency levels, when using VMware vStorage APIs for Data Protection with vSphere 4.1 for Windows/SQL Server backups.

Not Supported:

Microsoft does not support this combination of Windows and SQL Server. A combination unsupported by Microsoft.

Crash Consistent:

This is the state in which a system would be found after a system failure or power outage.

Application Consistent:

This is the state in which all databases are in-synch and represent the true status of the application.  In the table above, the light green represents a software snapshot provider (Windows 2003) and the dark green represents a hardware snapshot provider (Windows 2008).

As with all things backup, you should always verify the restores periodically to ensure the intended result can/will be achieved.

Note: This information is valid as of May 17, 2011 for VMware vStorage APIs for Data Protection and vSphere 4.1 editions.

As always, thanks for reading.

Brian

The requirement:
A requirement comes down for a set of 4 new virtual machines. Each of the VMs is 185 GB in size and will require a full nightly image backup.

The problem:
This additional 740 GB of daily backups is impacting the backup window and using a lot of storage in the process.  With no deduplication, a week's worth of all four VMs requires over 5TB of storage.  Each VM backup was taking 50 minutes to complete, and a total of 3 hours and 20 minutes each night was used just for these 4 VMs.

The resolution:
Use NetBackup 7 (which leverages the vStorage APIs for Data Protection) and VMware's Changed Block Tracking (CBT) functionality to reduce both the backup window and storage requirements.  The idea is to take a full image backup of each VM once a week, and then perform a block level incremental backup (BLIB) for the remaining 6 days.

The math:

The full backup on Saturday still takes the same amount of time as before, but on Sunday the block level incremental backup takes just 5 minutes to complete.  Monday - Friday offer similar results, and no single BLIB ever takes longer than 12 minutes to complete.  This means that I can get what is essentially a FULL backup of all four VMs in the time it used to take to get just one.  Notice also the amount of data that is backed up.  The entire set of BLIBs for 6 days use less space than a single FULL backup.  Both of these savings are very real and also very easy to implement.

The implementation:
Note: The VM must be running Virtual Machine Version 7, in order to use CBT, and the VM must be powered off to enable the CBT changes detailed below.  The first step is to enable CBT on a per-VM basis.  This is done via the "Edit Settings" for the VM.  Next choose the "Options" tab and scroll down to "Advanced" and select the "General" option.  Click the "Configuration Parameters" button and then add the following lines:

Name: ctk.Enabled

Value: true

Name: scsi0:0.ctkEnabled

Value: true

The scsiX:X.ctkEnabled configuration parameter should be added for each disk in the VM, and its value should reflect the Virtual Device Node for each disk.  For example, say there were two disks in the VM with Virtual Device Nodes scsi0:0 and scsi0:1 - there would need to be a configuration parameter added for each disk.  Each disk would be added as a new line.

The final configuration parameter (Name: disk.EnableUUID | Value: true) shown in the screenshot is optional.  It is used to enable application-consistent quiescing in Windows 2008 VMs and is discussed in kb 1028881.  Now might be a good time to make this change as well.

Once all of the name value pairs have been added to the Configuration Parameters, click OK twice and then power the VM up.  CBT is now enabled, and can be verified with the instructions provided in kb 1020128.

Note: If CBT is planned to be used for all virtual machines in the environment, it might make sense to modify any templates to include it by default.

Now that CBT is set up, the NetBackup policy can be set up to leverage BLIB.  Basically, the idea here is to set up the policy's schedule for one weekly full FlashBackup-Windows backup and 6 differential incremental backups.  First, the policy must be set up to use CBT.  This is configured in the "Attributes" tab for the policy and is accomplished by selecting the checkbox for "Perform block level incremental backups."

The final step is to configure the scheduling for this policy.  To do this, select the "Schedules" tab and create two new schedules.  The first schedule will be for a full weekly backup to be performed on Saturday.  The second schedule will be for a daily differential incremental backup to be performed Sunday through Friday.

Add the client that CBT was enabled on in the previous steps to this policy, and the policy is complete and ready to be used.

After the first full and differential incremental backups are both complete, the results can be verified with the "Backup Archive Restore" NetBackup application.  After selecting the desired VM, the differential backups will show up just like they would for a normal MS-Windows type policy.

The conclusion:

The combination of BLIB with CBT is a very effective solution for lowering backup windows and saving on storage.  For those that are already using vSphere and NetBackup 7, there is no additional licensing or cost associated with implementing either of these technologies.  The information presented here is a very basic approach to implementing BLIB and CBT, and there is much more information about this topic in both the "Designing Backup Solutions for VMware vSphere" Technical Note and the "Symantec NetBackup for VMware Administrator's Guide."

As always, thanks for reading!

Brian

I recently had to update the BIOS and iDRAC firmware on a Dell server running ESXi. I did not have the Dell Server Updates DVDs or the time required to download them, and instead used the Dell OMSA CentOS LiveCD and the Update Package for Red Hat Linux files option.  While the steps below are Dell specific, they could be modified very easily to work with just about any hardware vendor.

01. Download the Dell OMSA CentOS LiveCD, and make sure to read the README file.

02. Download the required updates from the Dell support site. Be sure to choose the "Update Package for Red Hat Linux" .BIN format for the downloads.  Again, make sure to read any notes or installation instructions included with the downloads.  For this set of Dell updates, the iDRAC firmware had to be updated prior to the BIOS.

03. Use ISO Master (or equivalent) to inject the .BIN files to the root of the Dell OMSA CentOS LiveCD ISO image, and then burn the image to disc. The updates could also be copied to a USB drive and then mounted from within the LiveCD.

04. After evacuating all VMs from the ESXi host and putting it in maintenance mode, boot the ESXi host to the Dell OMSA CentOS LiveCD.

05. Choose the "Boot (run from CD)" option at the menu.

06. Wait for the GUI to start, and then issue the following commands from the terminal window:

cd /mnt/live
./<NAME-OF-UPDATE.BIN>

07. When the update is complete, issue a reboot command from the GUI of the LiveCD or type the following command in the terminal window:

reboot

08. After the LiveCD completes shut down, remove it from the ESXi host.  Verify that the updated firmware versions are correct, and then test your ESXi host to ensure everything is working as expected.

Thanks for reading!

I recently ran into an issue upgrading a customer's vSphere environment.  This environment had 4 older servers running ESX 4.0 in a single cluster that were to be replaced with new hardware running ESXi 4.1. The upgrade process was basically to enable EVC on an existing cluster, remove one host at a time and finally add one host at a time back to the cluster.  When it was time to add the first host back into the existing cluster, I received the error "Cannot complete the configuration of the HA agent on the host.  Misconfiguration in the host network setup."  A quick search of the VMware Knowledge Base turned up kb 1019200 - "Configuring VMware High Availability fails with the error: Cannot complete the configuration of the HA agent on the host"

The kb article states that "This issue occurs if all the hosts in the cluster do not share the same service console or management network configurations. Some hosts may have service consoles using a different name or may have more service consoles than other hosts."  This makes perfect sense, as the ESX 4.0 hosts are using a Service Console connection type for the management network and the new ESXi 4.1 host is using a VMkernel connection type.  Fortunately, the kb article went on to discuss how to use the "das.bypassNetCompatCheck" option for HA to ignore this mismatched management connection condition.  Basically, turning off HA for the cluster and then turning it  on again with the "das.bypassNetCompatCheck" option allowed the ESXi 4.1 host to join the existing cluster without incident.  Once the fourth and final host was added to the cluster and the cluster contained only ESXi 4.1 hosts, the "das.bypassNetCompatCheck" was removed using the same procedure.

Hopefully this information will help others, as I am certain this upgrade scenario is a common one.

Thanks for reading,

Brian