Unable to start guests on NFS datastore on ESXi 5.x

[Nordmats]

Hello

We have a few ESXi 5.x hosts that are connected to a Linux (centOS 6 with SELinux enforcing) NFS server. The NFS traffic goes over a dedicated network.

We have two separate NFS shares that we connect as datastores on the machines.

Yesterday we were not able to power on any guest, old or newly created, that were stored on the NFS datastore. Local guests were fine.

This is the error in the guest logs:

2014-02-05T12:37:09.184Z| vmx| I120: CreateVM: Swap: generating normal swap file name.

2014-02-05T12:37:09.185Z| vmx| I120: Swap file path: '/<machine name>/./<machine name>-9f91d6fc.vswp'

2014-02-05T12:37:09.185Z| vmx| I120: VMXVmdb_GetDigestDiskCount: numDigestDisks = 0

2014-02-05T12:37:09.187Z| vmx| I120: Msg_Post: Error

2014-02-05T12:37:09.187Z| vmx| I120: [vob.sharedarea.regionlookup.symbolnotfound] SharedArea: Unable to find 'testSharedAreaPtr' in SHARED_PER_VM_VMX area.

2014-02-05T12:37:09.187Z| vmx| I120:

2014-02-05T12:37:09.187Z| vmx| I120: [vob.swap.poweron.createfailure.status] Failed to create swap file '/<machine name>/./<machine name>-9f91d6fc.vswp' : Not found

2014-02-05T12:37:09.187Z| vmx| I120: [msg.vmmonVMK.creatVMFailed] Could not power on VM : Not found.

2014-02-05T12:37:09.187Z| vmx| I120: [msg.monitorLoop.createVMFailed.vmk] Failed to power on VM.

2014-02-05T12:37:09.187Z| vmx| I120: ----------------------------------------

2014-02-05T12:37:09.194Z| vmx| I120: Module MonitorLoop power on failed.

This happened on two different ESXi hosts, one 5.1 and one 5.5. They had different NFS shares mounted.

No errors were found in audit log or iptables log on linux host.

We were able to get it to work again by unmounting the NFS shares and mounting them again. But we had to do it for both server.

Does anyone know why this might have happened and what could be done to prevent it from happening again?

Thanks

[schepp]

Hi,

Nordmats wrote:

Yesterday we were not able to power on any guest, old or newly created, that were stored on the NFS datastore.

newly created means you were still able to write on that datastores from the ESXi hosts?

Have you checked on the NFS Server if the new VMs were really stored there?

Regards

[abhilashhb]

Do you have free space on the volume? It says the swap file creation failed. Every time a VM is powered on there is a SWAP file created with the amount of RAm assigned to the VM. If the swap file creation fails, the VM will not power on. Can you confirm there is free space on the host?

Also, this might help

Knowledge Base #71

[zXi_Gamer]

Well, it seems that the guests' vmdk are in one datastore and the swap storage are in another datastore. the latter is not accessible. Since we are talking about NAS, can you provide esxcfg-nas -l

and

esxcli sched swap system  get

[Nordmats]

Hello

There was enough free space on both NFS datastore and on local datastore for swap. I checked that the machine was created on the NFS datastore.

I tried to copy the machine that I created on the NFS to the local datastore and power it on there. That worked.

I tried from SSH on the ESXi host to touch the .vswp file on the NFS datastore that it said that it couldn't create and that also worked.

~ # esxcfg-nas -l

DevTest NFS Datastore on <NFS Server> is /shares/devtest_nfs_vm_ds from <IP of NFS Server> mounted available

~ # esxcli sched swap system get

   Datastore Active: false

   Datastore Enabled: false

   Datastore Name:

   Datastore Order: 0

   Hostcache Active: false

   Hostcache Enabled: true

   Hostcache Order: 1

   Hostlocalswap Active: false

   Hostlocalswap Enabled: true

   Hostlocalswap Order: 2