[SOLVED! bug?] After update to 4.0u1: Failed to open 'XXXXX.vmdk': Broken pipe (2097161)

[jlangner]

Hello,

since having updated to ESX 4.0 update 1 (ESX-4.0.0-update01a.zip) we are having some serious problems with some of our virtual machines under ESX. Before the update we successfully shut down each virtual machine switched to maintainance mode, then run the update and rebooted the ESX server. It came up fine without any errors. However, some of our virtual machines can't be started anymore. E.g. the VI-Client only returns the following error for these clients:

Cannot open the disk '/vmfs/volumes/XXXXXX.vmdk' or one of the snapshot disks it depends on. Reason: Broken pipe.

After having checked the vmware.log file one can see that something strange is going on with the -delta.vmdk files:

Feb 01 09:18:23.802: vmx| DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
Feb 01 09:18:23.817: vmx| DISKLIB-CHAINESX : ChainESXOpenSubChainNode: can't create deltadisk node 419ae22a
-fwbps01-000001-delta.vmdk failed with error Broken pipe
Feb 01 09:18:23.818: vmx| DISKLIB-CHAIN : "/vmfs/volumes/49d9e5a3-fcef849b-2561-00144f0c9f73/fwbps01/fwbps0
1-000001.vmdk" : failed to open (Broken pipe).
Feb 01 09:18:23.819: vmx| DISKLIB-VMFS  : "/vmfs/volumes/49d9e5a3-fcef849b-2561-00144f0c9f73/fwbps01/fwbps0
1-000001-delta.vmdk" : closed.
Feb 01 09:18:23.820: vmx| DISKLIB-VMFS  : "/vmfs/volumes/49d9e5a3-fcef849b-2561-00144f0c9f73/fwbps01/fwbps0
1-s001.vmdk" : closed.
Feb 01 09:18:23.820: vmx| DISKLIB-LIB   : Failed to open '/vmfs/volumes/49d9e5a3-fcef849b-2561-00144f0c9f73
/fwbps01/fwbps01-000001.vmdk' with flags 0xa (Broken pipe).
Feb 01 09:18:23.821: vmx| DISK: Cannot open disk "/vmfs/volumes/49d9e5a3-fcef849b-2561-00144f0c9f73/fwbps01
/fwbps01-000001.vmdk": Broken pipe (2097161).

However, the CID chain seems to be fully intact:

root@fwbesx0 fwbps01--# grep -i cid *.vmdk
fwbps01-000001.vmdk:CID=73c9ab64
fwbps01-000001.vmdk:parentCID=66bfec3d
fwbps01.vmdk:CID=66bfec3d
fwbps01.vmdk:parentCID=ffffffff

Also the vmkfstools return a similar error when trying to clone the XXXX-00001.vmdk file:

root@fwbesx0 fwbps01--# vmkfstools -i ./fwbps01-000001.vmdk ./fwbps01-new.vmdk
Destination disk format: VMFS zeroedthick
Failed to open './fwbps01-000001.vmdk': Broken pipe (2097161).

However, when I try to clone the main base disk (without any -00000X) then the cloning works fine and without any problems. But of couse then everything is gone since the last snapshot was taken.

The only similarities I can spot with all these machines not running anymore is, that the base disk is a "XXXX-s001.vmdk" file rather than a "XXXX-flat.vmdk" file. So the base disk has the create type "twoGbMaxExtentSparse".

Here are the two description .vmdk files:

root@fwbesx0 fwbps01--# cat fwbps01.vmdk
# Disk DescriptorFile
version=1
CID=66bfec3d
parentCID=ffffffff
createType="twoGbMaxExtentSparse"

# Extent description
RW 33554432 SPARSE "fwbps01-s001.vmdk"

# The Disk Data Base
#DDB

ddb.toolsVersion = "8193"
ddb.longContentID = "9815e0ee51d4986f3c004e4166bfec3d"
ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "2088"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "ide"
ddb.encoding = "UTF-8"

root@fwbesx0 fwbps01--# cat fwbps01-000001.vmdk
# Disk DescriptorFile
version=1
CID=73c9ab64
parentCID=66bfec3d
createType="vmfsSparse"
parentFileNameHint="fwbps01.vmdk"
# Extent description
RW 33554432 VMFSSPARSE "fwbps01-000001-delta.vmdk"

# The Disk Data Base
#DDB

ddb.encoding = "UTF-8"
ddb.longContentID = "42d3f355c09cbb95ee0ef1fe73c9ab64"

And please find here the .vmx file of one of the machines not running anymore:

#!/usr/bin/vmware
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "7"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
nvram = "fwbps01.nvram"
deploymentPlatform = "windows"
virtualHW.productCompatibility = "hosted"
unity.customColor = "|23C0C0C0"
tools.upgrade.policy = "useGlobal"
powerType.powerOff = "soft"
powerType.powerOn = "default"
powerType.suspend = "hard"
powerType.reset = "soft"

displayName = "fwbps01"
extendedConfigFile = "fwbps01.vmxf"
floppy0.present = "TRUE"

memsize = "1024"
ide1:0.present = "TRUE"
ide1:0.clientDevice = "TRUE"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.clientDevice = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
guestOSAltName = "Microsoft Windows Server 2003, Standard Edition (32-Bit)"
guestOS = "winnetstandard"
uuid.location = "56 4d 48 9e cc 4b d3 ad-f9 e5 47 01 94 9e f4 58"
uuid.bios = "56 4d 48 9e cc 4b d3 ad-f9 e5 47 01 94 9e f4 58"
vc.uuid = "52 bc 44 4e ae ec 64 3d-4a de 6e dc ab 20 20 08"

floppy0.fileName = "/dev/fd0"
annotation = "Alter Printserver|0A"

ide0:0.present = "TRUE"
ide0:0.fileName = "fwbps01-000001.vmdk"

ethernet0.generatedAddress = "00:0c:29:9e:f4:58"
tools.syncTime = "TRUE"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/49d9e5a3-fcef849b-2561-00144f0c9f73/fwbps01/fwbps01-b8fec46e.vswp"
ide0:0.redo = ""
vmotion.checkpointFBSize = "16777216"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
ethernet0.generatedAddressOffset = "0"
vmci0.id = "-1801522088"
hostCPUID.0 = "0000000568747541444d416369746e65"
guestCPUID.0 = "0000000568747541444d416369746e65"
userCPUID.0 = "0000000568747541444d416369746e65"
hostCPUID.1 = "00100f230004080000802009178bfbff"
guestCPUID.1 = "00100f230000080080802001078bbbff"
userCPUID.1 = "00100f230004080000802001078bbbff"
hostCPUID.80000001 = "00100f230000038f000007ffefd3fbff"
guestCPUID.80000001 = "00100f230000038f000001e9ebd3bbff"
userCPUID.80000001 = "00100f230000038f000001e9ebd3bbff"
evcCompatibilityMode = "FALSE"
ide1:0.fileName = "/usr/lib/vmware/isoimages/windows.iso"

svga.autodetect = "TRUE"

sched.cpu.affinity = "all"
sched.mem.affinity = "all"

Any help to find out why after the update our virtual machines are not running anymore would be highly appreciated. The machines were really running fine right before the update and even other machines with a -flat.vmdk file and delta snapshots are working fine now. Only machines with sparse files and snapshots seem to be affected.

best regards,

jens

[jlangner]

After some more testing I finally got all my virtual machines running again. However, it seems to be a bug in ESX 4.0 update 1a that these XXXX-delta.vmdk files can't be used nor converted via vmkfstools. I have verified that by installing the old ESX 4.0-164009 version on another machine which had been used before my update to update 1a. Then I copied over the above mentioned vmdk files and run vmkfstools on it:

root@fwbesx1 esxrecover--# vmkfstools -i fwbps01-000001.vmdk ./fwbps01-new.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk 'fwbps01-000001.vmdk'...
Clone: 100% done.

As you can see the cloning/consolidating operation which previously didn't work on ESX 4.0u1 works fine with the old ESX 4.0 164009 build version. However, it remains unknown why this works with the old 164009 version but fails with ESX 4.0u1 (build 208167).

After having created the new consolidated vmdk file I copied it over to the new ESX 4.0u1 server and voila the virtual machine worked out-of-the-box.

However, if anyone has any idea how to investigate further I will keep a copy of one of the vmdk files which didn't convert vis 4.0u1. So I can perfectly supply that for bug checking or something similar.

best regards,

jens

[medmonds]

We too are having the same issue. We have been moving files from VMware server installs and converting them to ESX. The vmkfstools -i works fine on the older esx4 boxes, but not on the new ones and ONLY on the delta -000001 files do we have an issue. What we have been doing is running the conversion on the older ESX4 hosts then migrating them thru the VIC interface to the newer ones. That seems to work just fine. I think it may be an issue with the vmkfstools, not the file system.