StevenWE
Contributor
Contributor

Various File Operation Problems

I'm having quite a few problems with operations that act on files in my ESXi 5 environment and my Google foo is not returning anything useful.

* When I try to power on a VM (even one I just created), I get "Failed to create swap file '/sv01/./sv01-79fab788.vswp' : Not found"

I've been able to address this by adding "sched.swap.dir" to the vmx file but why is this occurring especially on a newly created VM?

* When I try to snapshot a VM, I get the following in the VMs logs:

vcpu-0| W110: FILE: File_Cwd: getcwd() failed: No such file or directory

vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to get full path name for 'sv01.vmx'.

vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx failed for file 'sv01.vmx': Error (3)

vcpu-0| I120: Checkpoint_Unstun: vm stopped for 31903 us

vcpu-0| I120: SCSI: switching scsi0 to push completion mode

vcpu-0| I120: Creating virtual dev for scsi0:0

vcpu-0| I120: DumpDiskInfo: scsi0:0 createType=11, capacity = 125829120, numLinks = 1, allocationType = 3

vcpu-0| I120: SCSIDiskESXPopulateVDevDesc: Using FS backend

vcpu-0| I120: DISKUTIL: scsi0:0 : geometry=7832/255/63

vcpu-0| W110: FILE: File_Cwd: getcwd() failed: No such file or directory

vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to get full path name for 'sv01.vmx'.

vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx failed for file 'sv01.vmx': Error (3)

vcpu-0| I120: SnapshotVMX_GetDictionaryBuffers: GetDictionaryBuffers failed. Error (3).

vcpu-0| I120: Vix: [10218 mainDispatch.c:6636]: Error VIX_E_FAIL in VMAutomation_AddSnapshotDictBuffersToPropList(): Unknown error

vcpu-0| I120: SnapshotVMXTakeSnapshotCB: Enter checkpoint status 1, mode 2.

vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'test': 0

vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Error (3).

vcpu-0| I120: Vix_TranslateSnapshotError: file IO error = 2

vcpu-0| I120: HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/808e6421-90cf4a4d/sv01/sv01.vmdk'

* When I try to migrate powered off VMs from local storage to a NFS store, I get "Error caused by file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk".  In addition, the following errors show up in the vpxa.log:

[30D70B90 info 'Libs' opID=0B209902-000006B4-28-79-7] FileIOErrno2Result: Unexpected errno=9, Bad file descriptor

[30D70B90 info 'Libs' opID=0B209902-000006B4-28-79-7] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/808e6421-90cf4a4d

/sv03/sv03.vmdk' : Bad file descriptor (589827).

[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk': Bad file descriptor (90003)

[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-LINK  : "/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk" : failed to open (Bad file descriptor).

[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-CHAIN : "/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk" : failed to open (Bad file descriptor).

[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-LIB   : Failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk' with flags 0x21f Bad file descriptor (589833).

[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFileDskOpenDisk: Failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk': Bad file descriptor (589833).

[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFile_Open: Open failed:

[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFile_GetInfo: Failed to open file

[30D70B90 error 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] Unable to get file info for file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk: The operation completed successfully

[30D70B90 verbose 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] NFC file error for file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk

[30D70B90 verbose 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] Closing NFC connection to server

[30D70B90 error 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcWorker] Error encountered while pre-processing copy spec for file ds:///vmfs/volumes/47355126-0eb7b838-cbd9-001c23cbbc96/sv03/sv03.vmdk:

--> vim.fault.FileFault

If I log-in to the environment via SSH, I get the same "Bad file descriptor" errors when trying to copy from the shell, however, I was able to use scp from a remote system to copy off the VM without error.

* When I try to host migrate a powered on VM on a NFS store I get the error, "An error occurred while saving the state for migration.  Unable to retrieve the current working directory: 0 (No such file or directory). Check if the directory has been deleted or unmounted.  Failed to start the virtual machine.  Module Migrate power on failed.  vMotion migration [-1062731769:1387777483531171] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout"

No amount of restarting the management agents or reboots have helped.  To me it appears that the whole environment is corrupt.



Does anyone have any suggestions for where to look for problems or how to correct?  The environment is over 1,300 miles away and I have to address all of these issues remotely.

Tags (3)
0 Kudos
2 Replies

I hope your are having issue in NFS data store. please check that u have enough space to create swap file and snapshot. and also Verify that the permissions of the NFS server have not been set to read-only for this ESX host. Verify that the NFS share was not mounted with the read-only box selected.

0 Kudos
StevenWE
Contributor
Contributor

Some might think I'm running a little lean on the NFS stores but there is plenty of space for swap files and one or two snapshots.

If space and/or permissions were an issue then setting the value of "sched.swap.dir" in the vmx files to the same NFS path where each VM is stored would not of fixed the "Failed to create swap file" problem.  Plus, I would not be able to even start VMs if permissions were an issue.

Here is the space usage of the NFS stores as seen from one of the ESX hosts:

# df -h | grep '/vmfs/volumes/nfsstore[12]'

NFS        200.0G 151.6G     48.4G  76% /vmfs/volumes/nfsstore1

NFS        150.0G 127.5G     22.5G  85% /vmfs/volumes/nfsstore2

Here is the portion of /etc/exports on the NFS server showing that write access from the ESX hosts is allowed:

/srv/vmware1  -sync,rw,no_root_squash,subtree_check  192.168.26.118 192.168.26.120

/var/vmware2  -sync,rw,no_root_squash,subtree_check  192.168.26.118 192.168.26.120

As you can see from the below NFS list nfsstore1 and nfsstore2 were not mounted read-only:

# esxcli storage nfs list

Volume Name  Host             Share               Accessible  Mounted  Read-Only  Hardware Acceleration

-----------  ----------------  ------------------  ----------  -------  ---------  ---------------------

Data         sv02.example.lan  /srv/data                 true     true       true  Not Supported

nfsstore1    sv02.example.lan  /var/opt/vmware           true     true      false  Not Supported

nfsstore2    sv02.example.lan  /srv/backup/vmware        true     true      false  Not Supported

To be sure there are not any write issues, I used touch to create a file on both NFS stores from one of the ESX hosts:

# touch /vmfs/volumes/nfsstore1/test

# touch /vmfs/volumes/nfsstore2/test

Plus, I just increased the size of both NFS stores to 250G but I'm still not able to snapshot a VM.

Also, It's worth mentioning that I got the same errors when trying to snapshot a VM on local storage.

0 Kudos