I'm having quite a few problems with operations that act on files in my ESXi 5 environment and my Google foo is not returning anything useful.
* When I try to power on a VM (even one I just created), I get "Failed to create swap file '/sv01/./sv01-79fab788.vswp' : Not found"
I've been able to address this by adding "sched.swap.dir" to the vmx file but why is this occurring especially on a newly created VM?
* When I try to snapshot a VM, I get the following in the VMs logs:
vcpu-0| W110: FILE: File_Cwd: getcwd() failed: No such file or directory
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to get full path name for 'sv01.vmx'.
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx failed for file 'sv01.vmx': Error (3)
vcpu-0| I120: Checkpoint_Unstun: vm stopped for 31903 us
vcpu-0| I120: SCSI: switching scsi0 to push completion mode
vcpu-0| I120: Creating virtual dev for scsi0:0
vcpu-0| I120: DumpDiskInfo: scsi0:0 createType=11, capacity = 125829120, numLinks = 1, allocationType = 3
vcpu-0| I120: SCSIDiskESXPopulateVDevDesc: Using FS backend
vcpu-0| I120: DISKUTIL: scsi0:0 : geometry=7832/255/63
vcpu-0| W110: FILE: File_Cwd: getcwd() failed: No such file or directory
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to get full path name for 'sv01.vmx'.
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx failed for file 'sv01.vmx': Error (3)
vcpu-0| I120: SnapshotVMX_GetDictionaryBuffers: GetDictionaryBuffers failed. Error (3).
vcpu-0| I120: Vix: [10218 mainDispatch.c:6636]: Error VIX_E_FAIL in VMAutomation_AddSnapshotDictBuffersToPropList(): Unknown error
vcpu-0| I120: SnapshotVMXTakeSnapshotCB: Enter checkpoint status 1, mode 2.
vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'test': 0
vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Error (3).
vcpu-0| I120: Vix_TranslateSnapshotError: file IO error = 2
vcpu-0| I120: HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/808e6421-90cf4a4d/sv01/sv01.vmdk'
* When I try to migrate powered off VMs from local storage to a NFS store, I get "Error caused by file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk". In addition, the following errors show up in the vpxa.log:
[30D70B90 info 'Libs' opID=0B209902-000006B4-28-79-7] FileIOErrno2Result: Unexpected errno=9, Bad file descriptor
[30D70B90 info 'Libs' opID=0B209902-000006B4-28-79-7] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/808e6421-90cf4a4d
/sv03/sv03.vmdk' : Bad file descriptor (589827).
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk': Bad file descriptor (90003)
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-LINK : "/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk" : failed to open (Bad file descriptor).
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-CHAIN : "/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk" : failed to open (Bad file descriptor).
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-LIB : Failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk' with flags 0x21f Bad file descriptor (589833).
[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFileDskOpenDisk: Failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk': Bad file descriptor (589833).
[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFile_Open: Open failed:
[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFile_GetInfo: Failed to open file
[30D70B90 error 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] Unable to get file info for file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk: The operation completed successfully
[30D70B90 verbose 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] NFC file error for file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk
[30D70B90 verbose 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] Closing NFC connection to server
[30D70B90 error 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcWorker] Error encountered while pre-processing copy spec for file ds:///vmfs/volumes/47355126-0eb7b838-cbd9-001c23cbbc96/sv03/sv03.vmdk:
--> vim.fault.FileFault
If I log-in to the environment via SSH, I get the same "Bad file descriptor" errors when trying to copy from the shell, however, I was able to use scp from a remote system to copy off the VM without error.
* When I try to host migrate a powered on VM on a NFS store I get the error, "An error occurred while saving the state for migration. Unable to retrieve the current working directory: 0 (No such file or directory). Check if the directory has been deleted or unmounted. Failed to start the virtual machine. Module Migrate power on failed. vMotion migration [-1062731769:1387777483531171] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout"
No amount of restarting the management agents or reboots have helped. To me it appears that the whole environment is corrupt.
Does anyone have any suggestions for where to look for problems or how to correct? The environment is over 1,300 miles away and I have to address all of these issues remotely.
I hope your are having issue in NFS data store. please check that u have enough space to create swap file and snapshot. and also Verify that the permissions of the NFS server have not been set to read-only for this ESX host. Verify that the NFS share was not mounted with the read-only box selected.
Some might think I'm running a little lean on the NFS stores but there is plenty of space for swap files and one or two snapshots.
If space and/or permissions were an issue then setting the value of "sched.swap.dir" in the vmx files to the same NFS path where each VM is stored would not of fixed the "Failed to create swap file" problem. Plus, I would not be able to even start VMs if permissions were an issue.
Here is the space usage of the NFS stores as seen from one of the ESX hosts:
# df -h | grep '/vmfs/volumes/nfsstore[12]'
NFS 200.0G 151.6G 48.4G 76% /vmfs/volumes/nfsstore1
NFS 150.0G 127.5G 22.5G 85% /vmfs/volumes/nfsstore2
Here is the portion of /etc/exports on the NFS server showing that write access from the ESX hosts is allowed:
/srv/vmware1 -sync,rw,no_root_squash,subtree_check 192.168.26.118 192.168.26.120
/var/vmware2 -sync,rw,no_root_squash,subtree_check 192.168.26.118 192.168.26.120
As you can see from the below NFS list nfsstore1 and nfsstore2 were not mounted read-only:
# esxcli storage nfs list
Volume Name Host Share Accessible Mounted Read-Only Hardware Acceleration
----------- ---------------- ------------------ ---------- ------- --------- ---------------------
Data sv02.example.lan /srv/data true true true Not Supported
nfsstore1 sv02.example.lan /var/opt/vmware true true false Not Supported
nfsstore2 sv02.example.lan /srv/backup/vmware true true false Not Supported
To be sure there are not any write issues, I used touch to create a file on both NFS stores from one of the ESX hosts:
# touch /vmfs/volumes/nfsstore1/test
# touch /vmfs/volumes/nfsstore2/test
Plus, I just increased the size of both NFS stores to 250G but I'm still not able to snapshot a VM.
Also, It's worth mentioning that I got the same errors when trying to snapshot a VM on local storage.