VMware Cloud Community
thoberre
Contributor
Contributor

Unable to start VM : Invalid argument on *-flat.vmdk

Hi,

I have an issue on a VMFS5 volume (ESXi5) where the guest vmdk files looks like they are locked in some way.

The guest does not power on (see vmware.log below) and every other command accessing the flat-files fails with an Invalid argument error, including vmfstools and even touch. mv-ing works however, but the same error remains after renaming and changing pointers in the files.

This looks to affect both vmdk disks/files on one particular guest on one volume - but does not affect other guests on same volume nor other vdisks for the same guest on other volumes. (The guest have 4 disks attached).

I don't know how this happened, but it happened at the same time as one of my backup disks failed as a freak coincidence (or some unknown external/environmental issue). I actually lost both my live disks and the backup the same night - it's only a development/test server - but still a significant amount of work lost if not solvable.

I've seen others report on similar errors, but have not been able to understand if this is solvable or how. Really appreciate any pointer in the right direction! Smiley Happy

Guest vmware.log:

2012-06-29T22:30:22.795Z| vmx| DISK: OPEN scsi1:0 '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk' independent-persistent R[]

2012-06-29T22:30:22.830Z| vmx| FileIOErrno2Result: Unexpected errno=22, Invalid argument

2012-06-29T22:30:22.830Z| vmx| AIOGNRC: Failed to open '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest-flat.vmdk' : Invalid argument (1600000002) (0x2013).

2012-06-29T22:30:22.830Z| vmx| DISKLIB-VMFS  : "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest-flat.vmdk" : failed to open (Invalid argument): AIOMgr_Open failed. Type 3

2012-06-29T22:30:22.830Z| vmx| DISKLIB-LINK  : "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk" : failed to open (Invalid argument). 

2012-06-29T22:30:22.830Z| vmx| DISKLIB-CHAIN : "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk" : failed to open (Invalid argument).

2012-06-29T22:30:22.830Z| vmx| DISKLIB-LIB   : Failed to open '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk' with flags 0xa Invalid argument (1441801).

2012-06-29T22:30:22.830Z| vmx| DISK: Cannot open disk "/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk": Invalid argument (1441801).

2012-06-29T22:30:22.830Z| vmx| Msg_Post: Error

2012-06-29T22:30:22.830Z| vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4ea9f2f3-7f904f43-3665-001b212e70ba/guest/guest.vmdk' or one of the snapshot disks it depends on.

2012-06-29T22:30:22.830Z| vmx| [msg.disk.configureDiskError] Reason: Invalid argument.

2012-06-29T22:30:22.830Z| vmx| ----------------------------------------

2012-06-29T22:30:22.838Z| vmx| Module DiskEarly power on failed.

2012-06-29T22:30:22.838Z| vmx| VMX_PowerOn: ModuleTable_PowerOn = 0

Reply
0 Kudos
9 Replies
avarude
Contributor
Contributor

Reply
0 Kudos
thoberre
Contributor
Contributor

Hi,

tanks for the reply. I have done investigations according to that article. I have rebboted the ESXi server, and made sure that there is no vm's running on the host.

What I have found however is that the vmkfstools -D guest-flat.vmdk reports that the file is locked by a mac address of an inactive/down interface on the host. I'm unable to release this lock - is there any way of releasing such a lock?

Reply
0 Kudos
fr0nk
Contributor
Contributor

You break a lock with -B (Breaklock) iirc.


The error message seems type of confusing. You're dealing with an independent disk (which isn't affected by a snapshot) and the log is complaining about not being able to open the disk OR "one of the snapshot disks it depends on". So it's the disk.

Reply
0 Kudos
thoberre
Contributor
Contributor

hmm, the lock is apparently not on the device, but on the VMDK image.

I have done som additional research and found some disturbing errors in the vmkernel.log regarding missing heartbeat on vmfs volume - possibly corrupted disk. (not able to get to the server atm since it's booted into a liveCD recovery with vmfs-tools. Got the VMFS volument mounted - but have the same problems reading the vmdk-files here).

Talking about getting owned by Murphy on this one....... should not be possible to wreck two different boxes at the same time as this, loosing both prod and backup Smiley Sad

Reply
0 Kudos
continuum
Immortal
Immortal

> failed to open (Invalid argument): AIOMgr_Open failed. Type 3

In my experience the only way to get that vmdk is to use a Linux LiveCD .
If you have mounted the VMFS volume and are able to see the vmdk - try dd_rescue


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

admin
Immortal
Immortal

Hi thoberre

Welcome to the community.

If you have lost or damaged data on a virtual drive, you will

need special software to help you recover it. One such solution is a program

called MediaHeal Repair VMDK.

"Life is never easy for those who dream"
Reply
0 Kudos
continuum
Immortal
Immortal

that tool will not help - for this task you need a tool that can read VMFS


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
thoberre
Contributor
Contributor

For curious people, futher investigations makes med believe this is a datacorruption issue rather than a filelock problem. However I don't have enough knowledge about the inner workings of VMFS to pursuade this further (one hope was that the VMFS metadata files held information about filelocks that I could clear).

I will post results if I get to try the diskdump method, but it might be that this is too time consuming for two 2TB vmdk files Smiley Happy

Reply
0 Kudos
NavalgundRaj
Enthusiast
Enthusiast

Hi thoberre,

How many backup vm are there in environment and one vm which takes backup may also be causing this.

Go to Backup vm or any third party backvm if you have and go to edit setting see if there are any disk from this server , just remove the disk from the backup server and try powering on.

Note: If you found this correct or  answer useful please consider the use of the Correct buttons to award points.

Regards

Basavaraj.R Navalgund

Note: If you found this correct or answer useful please consider the use of the Correct buttons to award points. Regards Basavaraj.R Navalgund
Reply
0 Kudos