Cannot complete file creation operation

[zdickinson]

We use Zerto to replicate our production VMs to our 3 Node vSAN 6 cluster in DR.  I can create new VMs, delete them, create folders, copy ISO files, etc...  However, when Zerto tries to do anything; test failover, create new protection group, etc..., I get "Cannot complete file creation operation".  And I see the following in vpxa.log on the host trying to do the operation:

--> Result:

--> (vim.fault.CannotCreateFile) {

-->    faultCause = (vmodl.MethodFault) null,

-->    faultMessage = (vmodl.LocalizableMessage) [

-->       (vmodl.LocalizableMessage) {

-->          key = "com.vmware.esx.hostctl.default",

-->          arg = (vmodl.KeyAnyValue) [

-->             (vmodl.KeyAnyValue) {

-->                key = "reason",

-->                value = "Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)"

-->             }

-->          ],

-->          message = "Operation failed, diagnostics report: Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)"

-->       }

-->    ],

-->    file = "Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)"

-->    msg = "Received SOAP response fault from [<cs p:1f31ce78, TCP:localhost:8307>]: CreateDirectory

--> Cannot complete file creation operation."

--> }

--> Args:

-->

--> Arg spec:

--> (vpxapi.VmLayoutSpec) {

-->    vmLocation = (vpxapi.VmLayoutSpec.Location) null,

-->    multipleConfigs = <unset>,

-->    basename = "Z-VRAH-gvvsan1.rpionline.com-586248",

-->    baseStorageProfile = <unset>,

-->    disk = (vpxapi.VmLayoutSpec.Location) [

-->       (vpxapi.VmLayoutSpec.Location) {

-->          url = "ds:///vmfs/volumes/vsan:fe0afcdab0b14afe-b39f91c71d1f1a76/1d00fa54-b4a3-9158-b876-ecf4bbcfd398/f79828c5-af94-4c14-8117-712276546bdd/vm-27637/Temo.vmdk",

-->          key = 16020,

-->          sourceUrl = <unset>,

-->          urlType = "exactFilePath",

-->          storageProfile = <unset>

-->       }

-->    ],

-->    reserveDirOnly = <unset>

Everything I look into leads me back to an MTU issue and this KB:  VMware KB: Creating new objects on a VMware Virtual SAN Datastore fails and reports the error: Faile...

I enabled the vDS health check and sure enough, it reports and MTU issue.  There is also this:

[root@gvvsan1:~] vmkping -d -s 1472 10.7.7.121

PING 10.7.7.121 (10.7.7.121): 1472 data bytes

1480 bytes from 10.7.7.121: icmp_seq=0 ttl=64 time=0.476 ms

1480 bytes from 10.7.7.121: icmp_seq=1 ttl=64 time=0.401 ms

1480 bytes from 10.7.7.121: icmp_seq=2 ttl=64 time=0.298 ms

--- 10.7.7.121 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.298/0.392/0.476 ms

[root@gvvsan1:~] vmkping -d -s 1473 10.7.7.121

PING 10.7.7.121 (10.7.7.121): 1473 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

--- 10.7.7.121 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

MTU size of 1472 results in a success and a size of 1473 results in a failure, the ping come back with a size of 1480.

My physical Dell 4032 switches have an MTU of 1518 and the vDS have an MTU of 1500.  My question is, what should they be and is there anywhere else I need to verify/set the MTU?

I have cases opened with Zerto, VMware, and my local networking consultant.  As always... thank you, Zach.

[ramakrishnak]

> message = "Operation failed, diagnostics report: Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)"

looks like osfs namespace directory creation is failing.

either Zerto is calling into wrong routine or our osfs dir creation is itself failing

Looking at /var/log/osfsd.log might give more insights

also am assuming that you can manually create vm dir from UI or using /usr/lib/vmware/osfs/bin/osfs-mkdir

Thanks

[zdickinson]

You are correct, I can create VMs through the UI.  I also noticed that v 6 requires a rev newer firmware for the LSI card I have.  I am going to make sure all drivers and firmware match the HCL for v 6 and not for v5.5 u2.  Will update when I have more information.  Thank you, Zach.

[zdickinson]

I have just noticed that the Micron P420M SSD is not on the vSAN 6 HCL.  In fact there are no PCIe cards on the vSAN 6 HCL.  Is that an oversight or true?  Thank you, Zach.

[ramakrishnak]

Certification plan for PCIE SSDs for VSAN 6.0 is in progress. hence this is not yet showing up in HCL

Thanks,

[DDunaway]

Any updates on this issue?

I'm having the exact same issue with another product, Unidesk.

I'm able to create VMs just fine with the vSphere Client or Web Client, but the Unidesk appliance attempts to create objects in the VSAN datastore and it fails.

I was also reading about MTU problems, but my vDS health check is all clear.

this worked just fine in the first iteration of VSAN. I only started having these problems after upgrading to ESXi 6 and vSAN 2.0

btw, I'm running on completely unsupported hardware (home lab)

Thanks!

-David

[zdickinson]

It turned out to be a problem with your replication solution.  They will not support vSAN 6 until their next update.  As to why they had the issue, I was not given that information.  Thank you, Zach.