Hi,
I have an issue on a VMFS5 volume (ESXi5) where the guest vmdk files looks like they are locked in some way.
The guest does not power on (see vmware.log below) and every other command accessing the flat-files fails with an Invalid argument error, including vmfstools and even touch. mv-ing works however, but the same error remains after renaming and changing pointers in the files.
This looks to affect both vmdk disks/files on one particular guest on one volume - but does not affect other guests on same volume nor other vdisks for the same guest on other volumes. (The guest have 4 disks attached).
I don't know how this happened, but it happened at the same time as one of my backup disks failed as a freak coincidence (or some unknown external/environmental issue). I actually lost both my live disks and the backup the same night - it's only a development/test server - but still a significant amount of work lost if not solvable.
I've seen others report on similar errors, but have not been able to understand if this is solvable or how. Really appreciate any pointer in the right direction!
Guest vmware.log:
2012-06-29T22:30:22.795Z| vmx| DISK: OPEN scsi1:0 '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk' independent-persistent R[]
2012-06-29T22:30:22.830Z| vmx| FileIOErrno2Result: Unexpected errno=22, Invalid argument
2012-06-29T22:30:22.830Z| vmx| AIOGNRC: Failed to open '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest-flat.vmdk' : Invalid argument (1600000002) (0x2013).
2012-06-29T22:30:22.830Z| vmx| DISKLIB-VMFS : "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest-flat.vmdk" : failed to open (Invalid argument): AIOMgr_Open failed. Type 3
2012-06-29T22:30:22.830Z| vmx| DISKLIB-LINK : "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk" : failed to open (Invalid argument).
2012-06-29T22:30:22.830Z| vmx| DISKLIB-CHAIN : "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk" : failed to open (Invalid argument).
2012-06-29T22:30:22.830Z| vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk' with flags 0xa Invalid argument (1441801).
2012-06-29T22:30:22.830Z| vmx| DISK: Cannot open disk "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk": Invalid argument (1441801).
2012-06-29T22:30:22.830Z| vmx| Msg_Post: Error
2012-06-29T22:30:22.830Z| vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk' or one of the snapshot disks it depends on.
2012-06-29T22:30:22.830Z| vmx| [msg.disk.configureDiskError] Reason: Invalid argument.
2012-06-29T22:30:22.830Z| vmx| ----------------------------------------
2012-06-29T22:30:22.838Z| vmx| Module DiskEarly power on failed.
2012-06-29T22:30:22.838Z| vmx| VMX_PowerOn: ModuleTable_PowerOn = 0
The vm could be hung.
check this link http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...
Hi,
tanks for the reply. I have done investigations according to that article. I have rebboted the ESXi server, and made sure that there is no vm's running on the host.
What I have found however is that the vmkfstools -D guest-flat.vmdk reports that the file is locked by a mac address of an inactive/down interface on the host. I'm unable to release this lock - is there any way of releasing such a lock?
You break a lock with -B (Breaklock) iirc.
The error message seems type of confusing. You're dealing with an independent disk (which isn't affected by a snapshot) and the log is complaining about not being able to open the disk OR "one of the snapshot disks it depends on". So it's the disk.
hmm, the lock is apparently not on the device, but on the VMDK image.
I have done som additional research and found some disturbing errors in the vmkernel.log regarding missing heartbeat on vmfs volume - possibly corrupted disk. (not able to get to the server atm since it's booted into a liveCD recovery with vmfs-tools. Got the VMFS volument mounted - but have the same problems reading the vmdk-files here).
Talking about getting owned by Murphy on this one....... should not be possible to wreck two different boxes at the same time as this, loosing both prod and backup
> failed to open (Invalid argument): AIOMgr_Open failed. Type 3
In my experience the only way to get that vmdk is to use a Linux LiveCD .
If you have mounted the VMFS volume and are able to see the vmdk - try dd_rescue
Hi thoberre
Welcome to the community.
If you have lost or damaged data on a virtual drive, you will
need special software to help you recover it. One such solution is a program
called MediaHeal Repair VMDK.
that tool will not help - for this task you need a tool that can read VMFS
For curious people, futher investigations makes med believe this is a datacorruption issue rather than a filelock problem. However I don't have enough knowledge about the inner workings of VMFS to pursuade this further (one hope was that the VMFS metadata files held information about filelocks that I could clear).
I will post results if I get to try the diskdump method, but it might be that this is too time consuming for two 2TB vmdk files
Hi thoberre,
How many backup vm are there in environment and one vm which takes backup may also be causing this.
Go to Backup vm or any third party backvm if you have and go to edit setting see if there are any disk from this server , just remove the disk from the backup server and try powering on.
Note: If you found this correct or answer useful please consider the use of the Correct buttons to award points.
Regards
Basavaraj.R Navalgund