VMware Cloud Community
KHBND
Contributor
Contributor
Jump to solution

Problem starting VM after problems witz snapshots and consolidation

OBJLIB-FILEBE : FileBEIoctl: ioctl operation failed on '/vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_1-000011-sesparse.vmdk' : Inappropriate ioctl for device (1638402)

Reply
0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

2020-08-22T15:13:47.230Z| vmx| I120: DISKLIB-LINK  : DiskLinkIsAttachPossible: Content ID mismatch (parentCID 589f6e8f != f4252530) /vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_2-000003.vmdk /vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_2.vmdk.

Sorry, I've missed this.

Changing the value for parentCID=589f6e8f to f4252530 in "Jobserver NEU_2-000003.vmdk" should do the trick.


André

View solution in original post

Reply
0 Kudos
18 Replies
a_p_
Leadership
Leadership
Jump to solution

Welcome to the Community,

please provide some more details:

  • explain - as detailed as possible - "after problems with snapshots and consolidation". What happened?
  • attach the VM's vmware.log to your next reply
  • run ls -lisa > filelist.txt from the command line in the VM's folder, and attach the filelist.txt to your next reply
  • what's the size of the datastore on which the VM is stored, and how much free disk space is available on it?

André

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Ok, I saw that - while I was writing my previous reply - you've already attached the vmware.log to your initial post.

What I found in the log is:

2020-08-22T14:19:10.495Z| Worker#0| I120: DISKLIB-LINK  : DiskLinkIsAttachPossible: Content ID mismatch (parentCID d8c345c2 != cea86101) /vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_1-000001.vmdk /vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_1.vmdk.

This happens if a parent virtual disk is modified.

Please enable SSH on the ESXi host, and use e.g. WinSCP to connect to it. Then download "Jobserver NEU_1.vmdk" as well as all the "Jobserver NEU_1-0000xx.vmdk" files (only the small descriptor files, i.e. those without flat, or sesparse in their file names). Once downloaded, compress/zip the files, and attach the .zip archive to your next reply.


André

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

Hello André,

thank's for your answer, I attached the files.

In my desperation, I had mounted the affected disks in another VM to see if I could still access the data.. That was probably a mistake

Karel

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

You didn't answer the other questions yet, so assuming that there's nothing unusual, you could try the following steps.

Edit - either via vi on the command line, or the built in editor in WinSCP - the "Jobserver NEU_1-000001.vmdk" file, and change the value for parentCID=d8c345c2 to cea86101 (which equals the CID in "Jobserver NEU_1.vmdk"). Then - before powering on the VM - take another snapshot, so that you can revert to the current state in case it's necessary.

Depending on what happened, and possible changes in "Jobserver NEU_1-flat.vmdk", you may see some file corruption. So be sure to verify the data before putting to VM into production again.

André

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

The size of the datastore is 21 TB - free size ist 9.7 TB

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

After changing the error is only for this volume '/vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_2-000005.vmdk

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

2020-08-22T15:13:47.230Z| vmx| I120: DISKLIB-LINK  : DiskLinkIsAttachPossible: Content ID mismatch (parentCID 589f6e8f != f4252530) /vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_2-000003.vmdk /vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_2.vmdk.

Sorry, I've missed this.

Changing the value for parentCID=589f6e8f to f4252530 in "Jobserver NEU_2-000003.vmdk" should do the trick.


André

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

Value of CID in this file is. Shoild I change it anyway?

CID=34989b5c

Sorry, parent CID is  right

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

Thank yu very much, Andrè. You save my live 😉

VM is up an running!

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

I still have problems with this VM. When I run a backup, the machine crashes. Even when I try to copy the data of the volumes in question with xcopy over a network share, the VM crashes. The VCenter says that the volumes need to be consolidated, but the process also crashes, with the error: Bad file descriptor.

I have added some files that might help.

Karel

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Bad file descriptor.

Does it say for with which file?

André

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

I tried consolidation again. This is the error, in german

Fehler bei Konsolidierung des Festplattenknotens 'scsi0:1': msg.snapshot.error-FAILED.

Fehler beim Senden der zugeteilten Bitmap an VMkernel während des Kopiervorgangs der Festplatten von '/vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_1-000014.vmdk' nach '/vmfs/volumes/577bd84d-10ac566a-84e9-a0369f8fe39c/Jobserver NEU/Jobserver NEU_1.vmdk'.

Karel

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Ok, I found

2020-08-22T23:20:27.558Z| Worker#0| I120: OBJLIB-FILEBE : FileBEIoctl: ioctl operation failed on '/vmfs/devices/deltadisks/332e1d85-Jobserver NEU_1-000014-sesparse.vmdk' : Bad file descriptor (589826)

in vmware-60.log.

Since you mentioned that you have more than 7 TB free disk space, it may be worth trying to clone, and consolidate the snapshots from the command line.

To do that, the VM has to be powered off, because we need full access to the .vmdk files, and - once cloning succeeds - manually edit its configuration file.

The steps I'm thinking of are:

  1. cleanly shut down the VM
  2. clone the affected virtual disk from the command line by running the following command from within the VM's folder:
    vmkfstools -i "Jobserver NEU_1-000014.vmdk" "Jobserver Clone_1.vmdk" -d zeroedthick
  3. if cloning succeeds, edit the VM's configuration (.vmx) file, and replace "Jobserver NEU_1-000014.vmdk" with "Jobserver Clone_1.vmdk"
  4. next, reload the VM by following steps 2 + 3 in https://kb.vmware.com/s/article/1026043​ so that the manual .vmx changes take effect

If everything works as expected, we can later delete the obsolete .vmdk files, and - if you want - rename the cloned .vmdk file back to it's original name.

Please note: Commands, and command line options are case-sensitive, and the quotes (Anführungszeichen) are required due to the space in the file names.

André

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

Okay, I'll try that. It's in progress.

Karel

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

Unsuccessful

Clone: 16% done.Failed to clone disk: Bad file descriptor (589833)

Karel

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

I'm actually not 100% sure whether it is the snapshot (sesparse) file which causes the error, or the base (flat) file.

Can you please run the clone command again, this time using the next file in the snapshot chain as the source, i.e.

vmkfstools -i "Jobserver NEU_1-000019.vmdk" "Jobserver Clone_1.vmdk" -d zeroedthick

If the target files ("Jobserver Clone_1.vmdk", and "Jobserver Clone_1-flat.vmdk") from the failed command still exist, please delete them prior to running the new command.

André

Reply
0 Kudos
KHBND
Contributor
Contributor
Jump to solution

The same result

Clone: 16% done.Failed to clone disk: Bad file descriptor (589833).

Karel

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

I'm afraid that it might be the base (flat) file which causes the issue.

In this case I'd suggest that you try to reach continuum​ via Skype (he speaks German).

André

Reply
0 Kudos