Adding more ESX hosts and upgrading at the same time. What's the preferred order of operations here?

[cypherx]

We are budgeting for more licensing and hardware for 2014 and want to do this right away in January.  I am trying to decide what is the best order of operations for adding two additional vmware hosts to our current vsphere installation, and also upgrading our vsphere / esx 4.1 u3 to the latest and greatest.

We have 3 Dell PowerEdge R710's running ESX 4.1 U3.  In this vmware cluster there is a vcenter server 4.1 U3 virtual machine managing this.  All three servers are about 90% memory utilized so we are adding 2 additional servers.  Likely the Dell PowerEdge R620 since we can fit 2 in the space of a 710.  The storage is on an EMC NX4 NFS san which will be upgraded also but not yet, let's just tackle the vmware part of it.

When adding 2 additional R620's, should I install ESX 4.1 on them and join them to the existing cluster, and THEN upgrade vcenter, and each esx one by one?  Reason I ask is because at 90% usage on the 3 current servers, I need the 2 additional servers to support operations while upgrades are taking place.  They would be able to hold vm's vmotioned off while each host is getting upgraded one at a time.

So is that the best way to go about adding hosts and upgrading?  Would I add them as 4.1's, then they would be available as vmotion targets so I have room to start upgrading hosts one at a time?

If I update vcenter to the latest version, it is backwards comparable and can manage the 4.1 hosts as well as 5.x hosts?  I suppose version 5.5 is the latest that I would be receiving from VMWare?  It's ESXi right, so how is that upgrade from ESX 4.1 to ESXi 5.5?  How much storage space do I need for ESXi 5.5, as I am specking out these Dell R620's and need to know how much hard drive to put in.

[gabinun · Accepted solution]

The Steps that you must follow

- Upgrade vCenter Server

- Upgrade ESX hosts (because we are doing a hardware refresh we are just going to install 5.5 on our new hosts and add them to the cluster and then decommission our existing)

- Upgrade VMWare Tools

- Upgrade Datastores

ESXi 5.5 has these storage requirements:

• Installing ESXi 5.5 requires a boot device that is minimum 1GB in size. When booting from a local disk or SAN/iSCSI LUN, a 5.2GB disk is required to allow the creation of the VMFS volume and a 4GB scratch partition on the boot device. If a smaller disk or LUN is used, the installer attempts to allocate a scratch region on a separate local disk. If a local disk cannot be found, the scratch partition (/scratch) is located on the ESXi host ramdisk, linked to /tmp/scratch. You can reconfigure /scratch to use a separate disk or LUN. For best performance and memory optimization, VMware recommends that you do not leave /scratch on the ESXi host ramdisk.
  
  
• To reconfigure /scratch, see Set the Scratch Partition from the vSphere Client in the vSphere Installation and Setup Guide.
  
  
• Due to the I/O sensitivity of USB and SD devices, the installer does not create a scratch partition on these devices. As such, there is no tangible benefit to using large USB/SD devices as ESXi uses only the first 1GB. When installing on USB or SD devices, the installer attempts to allocate a scratch region on an available local disk or datastore. If no local disk or datastore is found, /scratch is placed on the ramdisk. You should reconfigure /scratch to use a persistent datastore following the installation.
  
  
• In Auto Deploy installations, the installer attempts to allocate a scratch region on an available local disk or datastore. If no local disk or datastore is found /scratch is placed on ramdisk. You should reconfigure /scratch to use a persistent datastore following the installation.
  
  
• For environments that boot from a SAN or use Auto Deploy, it is not necessary to allocate a separate LUN for each ESXi host. You can co-locate the scratch regions for many ESXi hosts onto a single LUN. The number of hosts assigned to any single LUN should be weighed against the LUN size and the I/O behavior of the virtual machines.

[cypherx]

Thank you for your response.

Let me clarify and please point out if I am not understanding:

- Upgrade vCenter Server - Take existing 4.1 U3 snapshot it just to be sure, then upgrade it directly from 4.1 to 5.5.

  Because vCenter is backwards compatable, existing 4.1 hosts will still be managed in the time being.

- Add the two new Dell R620s with fresh ESXi 5.5 installs.  Join them to the new cluster with newly updated vCenter 5.5.

- Vmotion vm's from existing R710's to new 620s to prevent downtime.

- Upgrade existing R710's ESX 4.1 to ESXi 5.5

- Vmotion vm's back and distribute them evenly across all servers

- Upgrade vmware tools on each VM as time permits to allow for guest OS reboots.

- Upgrade datastores as time permits  (is any downtime required?)

[gabinun]

Yes.

Why switch to VMFS-5?

• Improved scalability and performance.
• Does not use SCSI-2 Reservations, but uses the ATS VAAI primitives.
• Uses GPT (GUID Partition Table) rather than MBR, allowing for pass-through RDM files greater than 2TB.
• VMFS5 in ESXi 5.5 now supports upto 62 TB VMDK and non-passthrough RDM. For more information, see Support for virtual machine disks larger than 2 TB in vSphere 5.5 (2058287).
• Newly created VMFS-5 datastores use a single block size of 1MB.
• Supports very small files (<1KB) by storing them in the metadata rather than in the file blocks.
• Uses sub-blocks of 8K rather than 64K, which reduces the space used by small files.
• Uses SCSI_READ16 and SCSI_WRITE16 cmds for I/O (VMFS-3 used SCSI_READ10 and SCSI_WRITE10 cmds for I/O).

What are the limitations for VMFS-5?

• VMFS-5 limits the number of extents to 32 and the total datastore size to 64TB, but the individual extents are not limited to 2TB each. For example, you can create a datastore with a LUN size of 64TB, or one with up to 32 extents up to maximum size of 64TB.
• Only pass-through RDMs (Raw Device Mapping) can be created with a size >2TB. Non-pass-through RDMs and virtual disk files are still limited to 2TB -512B in 5.0 and 5.1. In ESXi 5.5, support for non-passthrough RDMs have been increased to 62TB.
• Passthrough RDMs are supported up to 64TB in size.
• Both upgraded and newly-created VMFS-5 volumes supported the larger Passthrough RDM size.
• There is no space requirement for upgrading from VMFS3 to VMFS5.

Note: The actual maximum size of a LUN will depend on the capabilities of RAID controller/adapter driver that is used on the vSphere host.

For vSphere 5.x storage configuration maximums pertaining to 2TB LUN support please review Configuration Maximums for VMware vSphere 5.0 and also for Configuration Maximums for vSphere 5.1.

VMFS-5 Partitioning

Upgrading from VMFS-3 to VMFS-5

Can I upgrade while my virtual machines are running?

Yes. Upgrading from VMFS-3 to VMFS-5 can be done on-the-fly (virtual machines do not need to be powered-off, suspended, or migrated).
Do I have to use the command-line to upgrade to VMFS-5?

The upgrade to from VMFS-3 to VMFS-5 can be done either via the ESXi 5.x command-line or via vSphere Client.
Note: Ensure that all ESX hosts accessing the LUN are already on ESXi 5.x.

• To upgrade to VMFS-5 using the vSphere Client, Under Configuration > Storage, highlight the desired VMFS-3 datastore, and click onUpgrade to VMFS-5...
• To upgrade to VMFS-5 using the ESXi host command-line, use the command:
  
  # vmkfstools -T /vmfs/volumes/<VMFS3datastore>

My upgraded VMFS-5 does not have a 1MB block size. Why?
Upgraded VMFS-5 partitions will retain the partition characteristics of the original VMFS-3 datastore, including file block-size, sub-block size of 64K, etc. To take full advantage of all the benefits of VMFS-5, migrate the virtual machines to another datastore(s), delete the existing datastore, and re-create it using VMFS-5.

Note: Increasing the size of an upgraded VMFS datastore beyond 2TB changes the partition type from MBR to GPT. However, all other features/characteristics continue to remain same.

Troubleshooting VMFS-5 Upgrade Issues

The upgrade to VMFS-5 fails with these errors:

• There are hosts accessing this datastore which don't support VMFS-5
  
  To resolve this issue, ensure all ESXi hosts accessing the datastore are already running ESXi 5.x.
  
  
• An error occurred during host configuration. Operation failed, diagnostics report: Unable to Upgrade Filesystem: File system on device /vmfs/devices/disks/<device> cannot be online upgraded now because it is being used by some legacy host.
  
  To resolve this issue, check if any non-ESXi 5.x hosts (either standalone, or connected to another vCenter Server) is accessing this datastore, and has virtual machines running. If so, then run one of the:
  
  
  • Upgrade this host to ESXi 5.x.
  • vMotion all VMs accessing this datastore from the legacy ESX host to an ESXi 5.x host.
  • Storage vMotion, or Cold Migrate the virtual machines to a VMFS-3 filesystem.
  • Un-present the datastore from the legacy ESXi host. For more information see Removing a LUN containing a datastore from VMware ESXi/ESX 4.x (1029786).

[cypherx]

Thank you, that is good information on vmfs-5.

Currently our datastores are NFS on an EMC NX4.  We are also looking to add an EMC VNX5200.  Perhaps we will establish new vmfs-5 nfs datastores on the new system and vmotion machines to it.

Eventually the NX4 will be repurposed for backup / archival and the new VNX5200 will be in production.

[gabinun]

Ok Sorry.

Do you need any help or u r ok?

[cypherx]

Don't be sorry, that is all good information. 

Does the vmfs version apply if the datastore is NFS?

I think after that question I should be good and I will mark your answer.

Thank you again.

[gabinun]

No, as you know I was talking about VMFS.

Regards

[cypherx]

Ok thank you for your help today.  It provides a great high level overview of the milestones required to accomplish this upgrade.  Hopefully I will be ordering all the required pieces in the next few days.