VMware Cloud Community
Dthompson04
Contributor
Contributor

VCSA 6.7 U3 Designing VM Deployment File Location

I'm having a dispute on where the files for the VMs should be placed for a VCSA 6.7 implementation with four Hosts.  Should they be spread evenly between the hosts as currently deployed or on a single NAS appliance like recommended to our Configuration Manager from another department?

Now I've heard of storing VMs not on the host, but I don't fully understand that process and benefits.

  We've recently consolidated two VCSA clusters with two hosts each into a single cluster with four hosts.

From the beginning we've had a balance of VMs between the four Hosts and use vMotion when we needed to move the VMs for maintenance.

Another department is pushing us to get a NETAPP appliance to replace a three year old NAS.

My concern is if we store all of the VMs on the NETAPP what happens if it becomes unavailable?  Due to cost they want to get one NETAPP.  Below are several issues I have with this change of VM location.

1.  All VMs on a single point of failure.

2.  If all the VMs are on the NETAPP, do all the VMs use the devices network connections and resources?  I don't understand the operation of running the VMs on the Hosts and having the files on the NETAPP or NAS.

  Does the Host have to connect to the NAS/NETAPP to start the VM and is all this startup going across the network?

What is it that I'm missing of separating the host form the VM files?

Reply
0 Kudos
13 Replies
sjesse
Leadership
Leadership

I'm confused, VCSA usually means vcenter server appliance, and there is no clusters outside of setting up the vcenter high availability feature. Are you talking about the

VMware vSphere Storage Appliance (VSA) for Shared Storage

which is a few years past end of life.

Reply
0 Kudos
Dthompson04
Contributor
Contributor

Maybe I'm using the term Cluster wrong.  Is cluster when you have multiple ESXi hosts connected to a vCenter or is cluster when you have multiple vCenters?

Anyway, I'm talking about the management of four ESXi 6.7 hosts by a single vCenter, VCSA appliance.  I've worked with ESXi hosts starting with 5.5 and assisting with 4.1.

I've never worked with a vCenter until about a year ago so I'm short on best practices.

Reply
0 Kudos
Dthompson04
Contributor
Contributor

When Migrating VMs there are three options

Change compute resources only

Change storage only

Change both compute resource and storage.

I've always selected option 3 when migrating VMs because I'm going to be upgrading the opposite host.  The other team wants to split this with what I guess would be "Compute Resources" on the host and the "Storage" on the NETAPP.

Is this correct?

If so I have questions about how this functions.

1.  If all of the files are on the NETAPP does the host have to reach into the NETAPP to boot the VM?  Why would it be desirable to boot a VM separated from it's resources?

2.  Wouldn't this increase traffic over the network and in between the host and storage?

Reply
0 Kudos
sjesse
Leadership
Leadership

On the NetApp you have luns that you present to the esxi host over FC, ISCSI, or NFS. Each of these luns would get mounted and datastores would be created on them. These would be the same way the local datastores are used. All vm data including boot is stored in the device backing the datastore. The vm has no clue since its only addressing the vmdk files that are stored there.Really the best resource for your at this point is

https://www.netapp.com/us/media/tr-4597.pdf

its all of netapps best practices in relation to vsphere..

When Migrating VMs there are three options

Change compute resources only

Change storage only

Change both compute resource and storage.

I've always selected option 3 when migrating VMs because I'm going to be upgrading the opposite host.  The other team wants to split this with what I guess would be "Compute Resources" on the host and the "Storage" on the NETAPP.

Is this correct?

When you have shared storage like I mentioned above all the hosts can see the storage the same, which is required for the first 2 options

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

The other team wants to split this with what I guess would be "Compute Resources" on the host and the "Storage" on the NETAPP. Is this correct?

Yes, and unless you are pooling local storage using something like vSAN this is by FAR the most common deployment - has been for years and years...

If all of the files are on the NETAPP does the host have to reach into the NETAPP to boot the VM?  Why would it be desirable to boot a VM separated from it's resources?

Yes your hosts need to see that storage. As above, it’s been commonplace to separate compute and storage for years - central storage systems often use higher grade storage and have more resiliency.

Wouldn't this increase traffic over the network and in between the host and storage?

Yes, something you have to consider when designing and building out the environment.


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
ZibiM
Enthusiast
Enthusiast

Hello

Please consider that one of the fundamental reasons to deploy vCenter Server is the ability to organize your ESXi servers into HA & DRS clusters.

One of the very basic requirement for the HA cluster is to have some kind of shared storage that is accessed by all the ESXi servers that are member of the said HA cluster.

If your VMs are running on the shared storage (external like Netapp, internal like VSAN), then the failure of the ESXi server means it's load will be automatically restarted on the others that are part of the HA cluster.

Having VMs stored only on the single local datastore of the particular ESXi server means lengthy downtime, when said ESXi server will go down from hw, sw or network malfunction.

Yes the traffic is increased, but considering that the 2 links 10Gb are usually more than enough too run the load of the VMs, NFS storage, vMotion and backup of the average loaded server, this is not the problem. Especially now, when enterprises are moving from 10 Gb toward 25 Gb, and 10 Gb is becoming available even in the SoHo / Homelab area.

Usually storage units like the one from Netapp have embedded high availability features: 2 independent controllers working in active/standby, 2 separate interface cards (one for each controller), dual disk interfaces, etc. All these things make them bit more resilient than the local storage in the servers.

Reply
0 Kudos
Dthompson04
Contributor
Contributor

I've read about shared storage before, but didn't think you were supposed to put the VMs boot files on it.  We had another local discussion and now they are saying they want to put the VM and files on the NETAPP.

All of the connections are 10G ports on 1G network switches.  The only place I have 10G running is with a CAT6 cable between two hosts for HA.

It takes 4 hours to migrate a single VM from one host to another.  This is why I can't understand why it's better to have the files on the NETAPP.  If it takes 4 hours to move a VM from one host to another wouldn't it take the same amount of time to boot a VM between the host and storage?

Reply
0 Kudos
ZibiM
Enthusiast
Enthusiast

Not quite

When you copy the VM to the 2nd server, you just push all data it has.

When you launch this VM from network attached storage, you just copy the data this VM needs now to operate.

Please look on the performance graphs of your VMs related to the VM disks - how big data traffic you see ?

Usually this is not that high - VM needs lot of data transfer during boot, and during service startup, but after that most of the time whole load sits in the ESXi/VM OS memory and VM just reads additional data and writes results of the operations it processed.

Anyway - I'd rather avoid network attached storage if there is no 10G switching available.

The only viable option in such scenario is 2-node VSAN with directly attached network connections.

Reply
0 Kudos
sjesse
Leadership
Leadership

You should get 10gb, 4 hours is high, but 1gb in anything but a lab isn't enough in my opinion. As long as you don't have any latency in the network between the storage and the hosts the storage will work fine. If you set things up correctly you can have latency under 1ms even with very active vms, but that all depends on if the network between the storage and hosts isn't saturated and if the storage array itself responds in a reasonable time.

take a look at this too

https://docs.vmware.com/en/VMware-vSphere/6.7/vsphere-esxi-vcenter-server-67-storage-guide.pdf.pdf

it may be a bi to read but it may help clarify some things.

Reply
0 Kudos
Dthompson04
Contributor
Contributor

I understand FC and iSCSI as I've worked with it before.  Before the certification was retired I had my Storage Plus cert and understand how that works, I'm just concerned with the separation of boot files and resources.

To boot or reboot the VM the ESXi host has to pull the files across the network.  If a single NETAPP fails then all four hosts are separated from the VM storage.  In the event of a host reboot there are no VM files to load from.

I would think the better way of doing this would be to have duplicated storage between the ESXi host and Storage.  This way if the host goes down another host can access the files from Storage.  If the Storage goes down the host has it's local copy to run the VMs.

Reply
0 Kudos
Dthompson04
Contributor
Contributor

We have the Hosts and Storage on the same 4506 L3 switch right now and they are in the same VLAN.  There shouldn't be any delay.  Eventually when additional fiber gets installed between buildings two of the hosts will be in each building, but there will still be one NETAPP.

Reply
0 Kudos
sjesse
Leadership
Leadership

NetApp arrays general has come in an ha pair with either a floating wwpn for fiber channel or ip for iscsi or network. There is also probably and interconnect between them so if storage is available on one node and the other node requests it, that traffic flows between the interconnect. I think it would be in your best interest in ask NetApp about there HA capabilities to help with those questions. VSAN is closer to what your talking about where there are local disks and it replicates the data so its split across multiple hosts.

Architecture | vSAN Frequently Asked Questions (FAQ) | VMware

Reply
0 Kudos
sjesse
Leadership
Leadership

What model NetApp do you have?

Reply
0 Kudos