VMware Cloud Community
StanleyCup99
Contributor
Contributor
Jump to solution

Unable to create virtual machines in new datastore--frustrated beyond belief! :/

Hey all,

I am about to tear out the rest of my hair. I am hoping someone here has seen this behavior and can point me to a solution. I simply have tried everything I can think of, an nothing is making sense.

Summary: I have a Samsung 840 SSD, good for about 700GB, installed along with other SSDs in a box running ESXi 6.7

When I try to create a new VMFS6 datastore on this SSD, the process goes without issue. I have tried this under regular ESXi 6.7 UI, under ssh using command-line tools, and under vCenter 6.7 -- regardless, creating the datastore works just fine, as expected.

However, creating a new VM using that DS, or migrating an existing VM to that new DS, fails. EVERY TIME. The errors vary, they never seem to be the same, but it is always a combination of:

  • Unable to load configuration file '/vmfs/volumes/5ea9ef7c-23eb5578-3738-000af7a1d9e1/VM1/VM1.vmx'
  • An error occurred while creating a temporary dictionary file: Error.
  • Failed - The file is too large

No matter what though, the VM fails to be created/migrated. Google and other searches have yielded no help for this problem.

Of note: I can browse the new datastore, and upload files. I can create folders. I can ssh to the host and create files. I can create a text file using the text editor vi and read the file using cat. I can delete the files. In other words, the datastore seems to be perfectly happy. It just won't take virtual machines!

When I have tried creating the DS under command line, I have used partedUtil to create the partition (starting at sector 2048 as recommended for VMFS6) and then vmkfstools for creating the VMFS store. Currently, the SSD has a GPT format. I have tried taking the SSD into a Linux machine and clearing out all partitions there, and trying again under ESXi. Same result: the datastore is created but VMs can't be written to it. I have tried leaving a little space at the front of the partition and some space at the end of the partition. I did this because examining the other SSD's partitions, they seemed to have some free space before and after. I figured I would try that. No dice.

I truly am baffled. I don't know what could possibly be wrong with this datastore in terms on creating VMs or VMs being migrated to it, but being totally fine for writing files and directories directly or using the Browsing UI.

Here is a quick movie I made which shows the failed process, and you can also see I am displaying the info for the device in ESXi under ssh.

I am also attaching an image with the same information being shown in the ssh console.

If you would like any more details, I can provide them. If anyone can help, I will be forever grateful.

# polo

Device Details per ESXi.png

Tags (4)
Reply
0 Kudos
1 Solution

Accepted Solutions
StanleyCup99
Contributor
Contributor
Jump to solution

For those playing along at home--and those that find this in the next decade of Internet decay--the only way to get this to work was using VMFS5.

Why? That is a question to ponder another day. I have no idea. The same SSD had been used as a VMFS6 in an earlier installation of ESXi 6.7.x -- maybe something changed somewhere on the way to Update03…who knows.

Cheers!

View solution in original post

Reply
0 Kudos
7 Replies
continuum
Immortal
Immortal
Jump to solution

Do you see anything in the vmkernel.log of the ESXi that rejects loading your Test-VMs ?


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
StanleyCup99
Contributor
Contributor
Jump to solution

Thanks for pitching in!

I had put that project aside since it was driving me nuts, so I just started it all back up to give it another shot.

I got the SSD back in there and I will attempt the same stuff while I do a tail -f  /var/log/vmkernel.log and see if something else shines a light there.

I looked in the log when the ESXi box came up but it was just too full of stuff from just booting back up that it was a mess to grep.

I appreciate your help. I will get back to ya sometime before the end of the day.

Gotta finish the "real work" for today before I dive into that one Smiley Happy

Cheers!

Reply
0 Kudos
StanleyCup99
Contributor
Contributor
Jump to solution

So, I decided to update the ESXi box to the latest v6.7.x release, but then had some issues with bad VIBs and stuff.

I decided it was just a good time to re-install the latest and greatest v6.7.x on a different SSD, and after installation and configuration of ESXi, I am back up.

Now, I figured that running a brand new install of ESXi 6.7.x on a brand new SSD boot device, I would see a different result with the silly datastore that doesn't want to accept virtual machines. Right? Nope. No joy.

At first I just went ahead and tried to create a new VM in the datastore, and had the same issues. I tracked the vmkernel.log for answers, and saw something odd.

Next, I decided to blow up that datastore, and re-create it under this new ESX install--wouldn't usually matter, but at this point, what the hell, right?

So, I formatted the SSD with VMFS-6 and tried again. No joy.

Saw the same issue in the vmkernel.log -- and while I don't understand why I am seeing that error (Google was not much help) -- at least we are getting somewhere. I have NO IDEA why it is saying something about NFS and file locking…it's a SATA-connected SSD!

Here is a cut and paste of the log, from the point I deleted the datastore (old name NewDataStore), and then re-created the datastore (new name NewDSTest) and then tried to create a VM in said datastore. I highlighted the important stuff in blue and red, below.

Let me know your thoughts, and thanks again for checking in!

2020-05-19T01:01:11.674Z cpu8:2099725 opID=9d3a37b3)World: 11943: VC opID esxui-4af7-b6e9 maps to vmkernel opID 9d3a37b3

2020-05-19T01:01:11.674Z cpu8:2099725 opID=9d3a37b3)NVDManagement: 1461: No nvdimms found on the system

2020-05-19T01:01:29.745Z cpu9:2099185 opID=7fe279ef)World: 11943: VC opID esxui-1420-b6f0 maps to vmkernel opID 7fe279ef

2020-05-19T01:01:29.745Z cpu9:2099185 opID=7fe279ef)LVM: 16781: File system '[NewDataStore, 5ea9ef7c-23eb5578-3738-000af7a1d9e1]' (LV 5ea9ef7c-0f448c28-45ae-000af7a1d9e1) un-mounted.

2020-05-19T01:02:26.334Z cpu2:2099217 opID=e2f330e7)World: 11943: VC opID esxui-61e3-b70b maps to vmkernel opID e2f330e7

2020-05-19T01:02:26.334Z cpu2:2099217 opID=e2f330e7)LVM: 4107: [naa.50025388500adb61:1] Device expanded (actual size 1465143297 blocks, stored size 1465109886 blocks)

2020-05-19T01:02:26.347Z cpu2:2099217 opID=e2f330e7)LVM: 4107: [naa.50025388500adb61:1] Device expanded (actual size 1465143297 blocks, stored size 1465109886 blocks)

2020-05-19T01:02:26.386Z cpu2:2099217 opID=e2f330e7)LVM: 4107: [naa.50025388500adb61:1] Device expanded (actual size 1465143297 blocks, stored size 1465109886 blocks)

2020-05-19T01:02:26.409Z cpu2:2099217 opID=e2f330e7)LVM: 4107: [naa.50025388500adb61:1] Device expanded (actual size 1465143297 blocks, stored size 1465109886 blocks)

2020-05-19T01:02:26.428Z cpu2:2099217 opID=e2f330e7)LVM: 4107: [naa.50025388500adb61:1] Device expanded (actual size 1465143297 blocks, stored size 1465109886 blocks)

2020-05-19T01:02:26.428Z cpu2:2099217 opID=e2f330e7)LVM: 10366: Device naa.50025388500adb61:1 doesn't support TRIM

2020-05-19T01:02:26.523Z cpu2:2099217 opID=e2f330e7)LVM: 10432: Initialized naa.50025388500adb61:1, devID 5ec33022-d8e789e8-dc45-000af7a1d9e1

2020-05-19T01:02:26.568Z cpu2:2099217 opID=e2f330e7)LVM: 13557: Deleting device <naa.50025388500adb61:1>dev OpenCount: 0, postRescan: False

2020-05-19T01:02:26.588Z cpu2:2099217 opID=e2f330e7)LVM: 10526: Zero volumeSize specified: using available space (750135542272).

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START lfb>

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 35, childMetaOff 4096, Bits/R 3, Aff/RC 840, Aff/R 52

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)1397 resources, each of size 8192

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)Organized as 11 CGs, 8 C/CG and 16 R/C

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)CGsize 1114112. 0th CG at 65536.

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END lfb>

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START sfb>

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 2a, childMetaOff 0, Bits/R 2, Aff/RC 216, Aff/R 1

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)715264 resources, each of size 0

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)Organized as 175 CGs, 8 C/CG and 512 R/C

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)CGsize 65536. 0th CG at 65536.

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)child with 1114112 parent CGsize, 8 parent C/CG and 16 parent R/C

2020-05-19T01:02:26.692Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END sfb>

2020-05-19T01:02:26.795Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START fdc>

2020-05-19T01:02:26.795Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 4, childMetaOff 0, Bits/R 1, Aff/RC 256, Aff/R 1

2020-05-19T01:02:26.795Z cpu2:2099217 opID=e2f330e7)16384 resources, each of size 8192

2020-05-19T01:02:26.795Z cpu2:2099217 opID=e2f330e7)Organized as 8 CGs, 8 C/CG and 256 R/C

2020-05-19T01:02:26.795Z cpu2:2099217 opID=e2f330e7)CGsize 16842752. 0th CG at 65536.

2020-05-19T01:02:26.795Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END fdc>

2020-05-19T01:02:26.798Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START pbc>

2020-05-19T01:02:26.798Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 4, childMetaOff 0, Bits/R 1, Aff/RC 0, Aff/R 1

2020-05-19T01:02:26.798Z cpu2:2099217 opID=e2f330e7)0 resources, each of size 65536

2020-05-19T01:02:26.798Z cpu2:2099217 opID=e2f330e7)Organized as 0 CGs, 8 C/CG and 0 R/C

2020-05-19T01:02:26.798Z cpu2:2099217 opID=e2f330e7)CGsize 65536. 0th CG at 65536.

2020-05-19T01:02:26.798Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END pbc>

2020-05-19T01:02:26.799Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START sbc>

2020-05-19T01:02:26.799Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 4, childMetaOff 0, Bits/R 1, Aff/RC 256, Aff/R 1

2020-05-19T01:02:26.799Z cpu2:2099217 opID=e2f330e7)5116 resources, each of size 65536

2020-05-19T01:02:26.799Z cpu2:2099217 opID=e2f330e7)Organized as 3 CGs, 8 C/CG and 256 R/C

2020-05-19T01:02:26.799Z cpu2:2099217 opID=e2f330e7)CGsize 134283264. 0th CG at 65536.

2020-05-19T01:02:26.799Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END sbc>

2020-05-19T01:02:26.803Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START pb2>

2020-05-19T01:02:26.803Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 4, childMetaOff 0, Bits/R 1, Aff/RC 32, Aff/R 1

2020-05-19T01:02:26.803Z cpu2:2099217 opID=e2f330e7)256 resources, each of size 65536

2020-05-19T01:02:26.803Z cpu2:2099217 opID=e2f330e7)Organized as 1 CGs, 8 C/CG and 32 R/C

2020-05-19T01:02:26.803Z cpu2:2099217 opID=e2f330e7)CGsize 16842752. 0th CG at 65536.

2020-05-19T01:02:26.803Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END pb2>

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)Res3: 10702: SDDir: type: 0x5, fileLength: 0x11000, numBlocks: 1

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)FS3: 183: <START jbc>

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)signature 72666d64, version 1, flags 0, childMetaOff 0, Bits/R 1, Aff/RC 8, Aff/R 1

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)128 resources, each of size 2097152

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)Organized as 4 CGs, 4 C/CG and 8 R/C

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)CGsize 67141632. 0th CG at 65536.

2020-05-19T01:02:26.805Z cpu2:2099217 opID=e2f330e7)FS3: 185: <END jbc>

2020-05-19T01:02:26.816Z cpu2:2099217 opID=e2f330e7)Vol3: 1684: Created VMFS-6.82 with config 0x6 on vol 'NewDSTest'

2020-05-19T01:02:26.862Z cpu2:2099217 opID=e2f330e7)WARNING: NFS: 1226: Invalid volume UUID 5ea9ef7c-0f448c28-45ae-000af7a1d9e1

2020-05-19T01:02:26.862Z cpu2:2099217 opID=e2f330e7)Vol3: 1299: Could not open device '5ea9ef7c-0f448c28-45ae-000af7a1d9e1' for volume open: No such target on adapter

2020-05-19T01:02:26.862Z cpu2:2099217 opID=e2f330e7)Vol3: 1299: Could not open device '5ea9ef7c-0f448c28-45ae-000af7a1d9e1' for volume open: No such target on adapter

2020-05-19T01:02:26.862Z cpu2:2099217 opID=e2f330e7)Vol3: 1299: Could not open device '5ea9ef7c-0f448c28-45ae-000af7a1d9e1' for volume open: No such target on adapter

2020-05-19T01:02:26.863Z cpu2:2099217 opID=e2f330e7)Vol3: 1299: Could not open device '5ea9ef7c-0f448c28-45ae-000af7a1d9e1' for volume open: No such target on adapter

2020-05-19T01:02:26.863Z cpu2:2099217 opID=e2f330e7)FSS: 6092: No FS driver claimed device '5ea9ef7c-0f448c28-45ae-000af7a1d9e1': No filesystem on the device

2020-05-19T01:02:26.917Z cpu2:2099217 opID=e2f330e7)WARNING: NFS: 1226: Invalid volume UUID 5ec33022-c8bfc8c0-df11-000af7a1d9e1

2020-05-19T01:02:26.963Z cpu2:2099217 opID=e2f330e7)LVM: 16770: File system '[NewDSTest, 5ec33022-e96e09c8-64ca-000af7a1d9e1]' (LV 5ec33022-c8bfc8c0-df11-000af7a1d9e1) mounted in 'rw' mode.

2020-05-19T01:02:27.350Z cpu21:2100355)WARNING: NFS: 1226: Invalid volume UUID naa.50025388500adb62:3

2020-05-19T01:02:27.405Z cpu21:2100355)FSS: 6092: No FS driver claimed device 'naa.50025388500adb62:3': No filesystem on the device

2020-05-19T01:02:27.475Z cpu18:2099218 opID=8ac2d861)World: 11943: VC opID esxui-6f03-b719 maps to vmkernel opID 8ac2d861

2020-05-19T01:02:27.475Z cpu18:2099218 opID=8ac2d861)VC: 4616: Device rescan time 214 msec (total number of devices 9)

2020-05-19T01:02:27.475Z cpu18:2099218 opID=8ac2d861)VC: 4619: Filesystem probe time 167 msec (devices probed 8 of 9)

2020-05-19T01:02:27.475Z cpu18:2099218 opID=8ac2d861)VC: 4621: Refresh open volume time 0 msec

2020-05-19T01:02:29.581Z cpu2:2097973)LVM: 16789: One or more LVM devices have been discovered.

2020-05-19T01:04:00.715Z cpu20:2099220 opID=d39e8bf9)World: 11943: VC opID esxui-a5e5-b766 maps to vmkernel opID d39e8bf9

2020-05-19T01:04:00.715Z cpu20:2099220 opID=d39e8bf9)Fil3: 8608: Retry 10 for caller Fil3_CreateFile (status 'Optimistic lock acquired by another host')

2020-05-19T01:05:36.987Z cpu8:2097931)DVFilter: 6054: Checking disconnected filters for timeouts

Reply
0 Kudos
StanleyCup99
Contributor
Contributor
Jump to solution

For those playing along at home--and those that find this in the next decade of Internet decay--the only way to get this to work was using VMFS5.

Why? That is a question to ponder another day. I have no idea. The same SSD had been used as a VMFS6 in an earlier installation of ESXi 6.7.x -- maybe something changed somewhere on the way to Update03…who knows.

Cheers!

Reply
0 Kudos
Veers
Contributor
Contributor
Jump to solution

You are not alone... I have the same beavior on a NFS datastore! I have a Windows Server who have a NFS share and 3 ESX connected to that share. The connection work fine, I can browse the datastore, copie file, but i CAN'T move a VM nor create a new one.

I have the same two first errors but also a "16 (Device or resource busy") error.

I also have that un vmkernel.log

2021-06-29T13:58:54.689Z cpu36:2100519 opID=84a85801)World: 11986: VC opID kp6w9yy3-466484-auto-9zxx-h5:70192262-e2-01-ab-1039 maps to vmkernel opID 84a85801
2021-06-29T13:58:54.689Z cpu36:2100519 opID=84a85801)WARNING: NFS41: NFS41FileDoRemove:5254: Could not remove "test" (task process failure): Directory not empty
2021-06-29T13:58:54.689Z cpu36:2100519 opID=84a85801)VmMemXfer: vm 2100519: 2340: Evicting VM with path:/vmfs/volumes/e0a14384-c3b0cc73-0000-000000000000/test/test.vmx
2021-06-29T13:58:54.689Z cpu36:2100519 opID=84a85801)VmMemXfer: 207: Creating crypto hash
2021-06-29T13:58:54.690Z cpu36:2100519 opID=84a85801)VmMemXfer: vm 2100519: 2354: Could not find MemXferFS region for /vmfs/volumes/e0a14384-c3b0cc73-0000-000000000000/test/test.vmx
2021-06-29T13:58:54.782Z cpu22:2100076 opID=919aa7f6)World: 11986: VC opID kp6w9yy3-466484-auto-9zxx-h5:70192262-e2-01-de-104a maps to vmkernel opID 919aa7f6
2021-06-29T13:58:54.782Z cpu22:2100076 opID=919aa7f6)VmMemXfer: vm 2100076: 2340: Evicting VM with path:/vmfs/volumes/e0a14384-c3b0cc73-0000-000000000000/test/test.vmx
2021-06-29T13:58:54.782Z cpu22:2100076 opID=919aa7f6)VmMemXfer: 207: Creating crypto hash
2021-06-29T13:58:54.782Z cpu22:2100076 opID=919aa7f6)VmMemXfer: vm 2100076: 2354: Could not find MemXferFS region for /vmfs/volumes/e0a14384-c3b0cc73-0000-000000000000/test/test.vmx
2021-06-29T13:59:01.939Z cpu40:2097307)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-06-29T13:59:01.939Z cpu40:2097307) min,KB max,KB minLimit,KB eMin,KB rMinPeak,KB name
2021-06-29T13:59:01.939Z cpu40:2097307)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-06-29T13:59:01.939Z cpu40:2097307) 204800 204800 -1 204800 204800 host/vim/vmvisor/config-file-tracker
2021-06-29T13:59:01.939Z cpu40:2097307)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 1092 73832 python.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 132 132 uwWorldStore.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 136 136 worldGroup.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 0 72212 uw.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 136 136 vsiHeap.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 264 792 pt.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 288 288 cartelheap.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 0 0 uwshmempt.6479686
2021-06-29T13:59:01.939Z cpu40:2097307) 0 -1 -1 136 136 uwAsyncRemapHeap.6479686
2021-06-29T13:59:01.939Z cpu40:2097307)------------ ------------ ------------ ------------ ------------ ------------------------------

😕

Reply
0 Kudos
Veers
Contributor
Contributor
Jump to solution

Just find out that it's work with NFS 3 but not with NFS 4.1...

Reply
0 Kudos
Kashmir1979
Contributor
Contributor
Jump to solution

Hi, I have a very unique issue.  I have a nested ESXi, the datastores are virtual QEMU. It can read an attached disk, format it VMFS and show it as a datastore.

However, I cannot:

- Upload anything to this datastore, the upload never goes above 0% and eventually fails.
- It fails via web-browser and SCP

I can however create a virtual machine assign it a Thick provisioned disk and it will create it ..no issue. I have tried:

- VMFS5 and VMFS6
- iSCSI Lun Datastore
- Raw disk 

Same issue everywhere. Has anybody got a clue what can be the issue..

Thanks

Reply
0 Kudos