VCSA file copy/move to vSAN Datastore freezes at 10%

[cyrus104]

I have 3x identical Supermicro machines each with 8c/16t, 64GB ram, 512GB ESXi Drive, 1TB Samsung 970 Pro NVME Cache Drive, 6.4TB Intel DC P4600 NVME Capacity drive. The is a fairly simple setup for my homelab on 1 switch with 2 VLANs.

I just installed esxi 7 on all three machines with the management interface in a vlan. It pulls the proper IP address from the static DHCP server.

When I try to move any file from a local node or my network NAS (NFS) to a folder in my vSAN, it gets to 10% pretty quickly and then just sites at 10%. I left it overnight and it hadn't moved for errored out.

I'm not sure where to look to troubleshoot this as I couldn't find any errors.

Thanks

[TheBobkin]

Hello cyrus104​,

How (e.g. WinSCP, vSphere Client datastore browser) and where (e.g. directly onto vsanDatastore 'root' directory or into a namespace) are you uploading what (e.g. ISO, -flat.vmdk)?

If you are moving large -flat.vmdk into a namespace folder, it may have run out of space as these have a max size of 255GB (though there are ways around this) as they are not intended for storing large amounts of data directly, instead they are intended to house vmdk descriptors that point to vSAN Objects (instead of pointing to -flat.vmdk).

Bob

[cyrus104]

I was trying to use the vSphere Client datastore browser and trying to copy the file to the root directory. I am uploading a large vmdk that I would eventually like to import into another VM. I want to build a VM around this vmdk but then have it be regularly stored on the vsan datastore for those benefits. I haven't seen a good guide on doing that yet.

Each ESXI host has a drive barely big enough for the VM I want to put there so I can try and import it as a virtual disk / or vsan object.

[a_p_]

Please see Upload Files or Folders to vSAN Datastores for how to upload .vmdk files to a vSAN datastore.

André

[TheBobkin]

Hello cyrus104​,

"trying to copy the file to the root directory"

Don't do that - upload it to a namespace folder, a dummy one can be created simply by creating a VM with no disks or using osfs-mkdir:

VMware Knowledge Base

"I am uploading a large vmdk that I would eventually like to import into another VM. I want to build a VM around this vmdk but then have it be regularly stored on the vsan datastore for those benefits. I haven't seen a good guide on doing that yet."

This is fairly trivial, just create the VM, add a hard disk, select 'use an existing disk' and select the vmdk you have uploaded to a namespace.

"Each ESXI host has a drive barely big enough for the VM I want to put there so I can try and import it as a virtual disk / or vsan object."

Do you mean a non-vSAN datastore attached to each host is not big enough or that the vmdk is bigger than the vSAN disks/Disk-group capacity per node?

Bob

[cyrus104]

TheBobkinTheBobkin​, thanks for helping me out.

I did create a VM with no disk and am using the copy/move function to put the new VM into that folder. So I think it's a namespace folder with the file size restriction.

I did use the vmware-vdiskmanager to convert the vmdk I had into a stream optimized one. The file size is around 230GB, so it's not small.

To do this I am going to my NFS datastore, selecting the file to copy, use the copy button, select the vsandatastore/new_vm_folder it's now stuck at 11%.

"Each ESXI host has a drive barely big enough for the VM I want to put there so I can try and import it as a virtual disk / or vsan object."

Do you mean a non-vSAN datastore attached to each host is not big enough or that the vmdk is bigger than the vSAN disks/Disk-group capacity per node?

I'm sorry, I didn't fully form my though very well. My idea was to copy the VM to the local datastore, build a vm around it there, finally use migrate to move it over to vsandatastore.

Thanks again for offering advice.

[TheBobkin · Accepted solution]

Hello cyrus104​

Okay, so if you currently have the VMs data stored as -flat.vmdk(s) (either on your NFS or on local-VMFS) and whatever method you are doing is just copying that as -flat.vmdk to the namespace on vsanDatastore then that will lead to issue. If the NFS datastore is attached to the hosts (or you have -flat.vmdk on VMFS) you may be able to use vmkfstools to clone the data to vsanDatastore and create them properly as vmdk Objects e.g.:

# vmkfstools -i "/vmfs/volumes/NFS-DS-Name/VM-Name/VM-Name.vmdk" "/vmfs/volumes/vsanDatastore-Name/VM-Name-Or-Dummy-Namespace/VM-Name.vmdk" -d thin -W vsan

Validate before doing the above that you are not trying to create it in a namespace you have already filled up with file data etc. .

In case we are misunderstanding your current situation and/or the above doesn't make sense, if you could share/PM the output of the current namespaces (both source on NFS and destinations on VMFS and/or vsanDatastore) by changing to the directories and running this in each one:

# ls -lah

# du -ah

Bob

[cyrus104]

TheBobkin​,

I want to thank you for all the help you have provided me in troubleshooting. As this is a homelab for learning and running numerous random projects, I have decided to go back to Proxmox w/ Ceph for my Hyper-converged solution. I continue to use vmware workstation for home and work on my local machine. While I know VMware is the giant and the best choice for full enterprises, I've spent more time trying to understand why I cannot easily copy over a vmdk and just have it work. I have also run into a week or more of frustration trying to figure out where some of the bottle necks are in my small setup. My system only have 64GB of ram and that works perfect for Proxmox w/ Ceph and XCP-ng but 2 of my hosts in this setup are running at over 50% ram usage. When I asked about this and provided my system details to other forum members, I was told this is to be expected. My last issue is that I haven't had any issues navigating the interface for any other hypervisor manager except for VCSA. I've used ESXi on individual hosts for awhile and didn't think it was complicated, hard to find settings, or troubleshoot interface or disk issues. Again I understand this product is meant for much larger environments and normally has more than one person working on the system at a time but I've seen numerous places where things could be radically simplified and still be perfectly functional.

The forum community here has been great and I'll still be come back for my VMware Workstation Pro questions.

Thanks again for all of your help.

[srodenburg]

At the VMware guys 'n gals:  I see this a lot, people trying WinSCP etc. to simply upload/download VM's to a vSAN Datastore (and get unexpected results).

This needs to be made clearer somehow. I don't know how to be honest. Maybe in the GUI. Putting it in the documentation is good.

We cannot save folks that don't read documentation, don't google and just click&try. Those are "revolver clickers" that cannot be helped. But for the rest of the admins out there, this vSAN specific "thing" must be communicated clearer. I have an email template which I simply copy & paste and send off as a reply when I get this very question again. It happens that often.

Just my 2 cents. Stay safe.

[srodenburg]

Cyrus104 wrote:

"Again I understand this product is meant for much larger environments and normally has more than one person working on the system at a time but I've seen numerous places where things could be radically simplified and still be perfectly functional."

Then make suggestions on "how". vSphere is very feature rich, very powerful and the GUI must accommodate that somehow. There is a LOT to accommodate. VMware does not produce a "light / simplified" version.

You could also ask the question "why does a vSAN Cluster node" take up so much RAM?  Answer:  When using vSAN, the ESXi Kernel is also the "Storage Controller" and needs to store vSAN meta-data (remember, vSAN is object-storage) and cache and that takes up a lot of memory space. Activate dedupe and compression and it needs even more. vSAN was not designed to be "small lab friendly". It plays with the big guns out there. Do you know how much RAM a medium size "classic" storage controllers needs for caching and metadata etc. etc.) A lot! Some of them have 64 or 128gig RAM for cache only, not to mention the rest.

Comparing vSphere and vSAN to Proxmox etc. is not wise. Very different market/target audience. Like comparing a small boat (ProxMox) to a battleship and then complaining that the bridge of the battleship has so many more buttons and switches.