ESXi 5.5 Update 3: Deleting Snapshot Crashes VM: Unexpected signal: 11.

[gregsn]

After upgrading from ESXi 5.5 Update 2 to Update 3, deleting snapshot randomly crashed virtual machines.  Updating VMware tools didn't help.  No crashing problems before with snapshots on the same server for the last ~3 years.  Problem started with Update 3.

So far, this has happened with Windows XP, 2003, 2008R2 OS with updated VMware Tools.

Here is an example of one of the systems that just crashed right after a snapshot delete:

2015-09-20T07:02:51.386Z| vcpu-0| I120: SnapshotVMXConsolidateOnlineCB: nextState = 4 uid 0

2015-09-20T07:02:51.386Z| vcpu-0| I120: Closing disk scsi0:0

2015-09-20T07:02:51.388Z| vcpu-0| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 4858240.

2015-09-20T07:02:51.388Z| vcpu-0| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2015-09-20T07:02:51.394Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : closed.

2015-09-20T07:02:51.394Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000001-delta.vmdk" : closed.

2015-09-20T07:02:51.394Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-flat.vmdk" : closed.

2015-09-20T07:02:51.395Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : open successful (24) size = 16912384, hd = 1909127. Type 8

2015-09-20T07:02:51.395Z| vcpu-0| I120: DISKLIB-DSCPTR: Opened [0]: "es-1.company.local-000002-delta.vmdk" (0x18)

2015-09-20T07:02:51.395Z| vcpu-0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002.vmdk' (0x18): vmfsSparse, 134217728 sectors / 64 GB.

2015-09-20T07:02:51.395Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2015-09-20T07:02:51.395Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 1909127, extentType = 0

2015-09-20T07:02:51.396Z| vcpu-0| I120: DISKLIB-LIB   : Resuming change tracking.

2015-09-20T07:02:51.396Z| vcpu-0| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 1909127.

2015-09-20T07:02:51.396Z| vcpu-0| I120: DISKLIB-CBT   : Successfuly created cbt node 16218d-cbt.

2015-09-20T07:02:51.396Z| vcpu-0| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/16218d-cbt

2015-09-20T07:02:51.397Z| vcpu-0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002.vmdk" (flags 0x18, type vmfsSparse).

2015-09-20T07:02:51.397Z| vcpu-0| I120: SnapshotVMXNeedConsolidateIteration: Size of helper disk '/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002.vmdk' = 17825792 bytes, approx. time required for consolidating helper disk = 0.447157 sec.

2015-09-20T07:02:51.398Z| vcpu-0| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 1909127.

2015-09-20T07:02:51.398Z| vcpu-0| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2015-09-20T07:02:51.406Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : closed.

2015-09-20T07:02:51.406Z| vcpu-0| I120: SnapshotVMXNeedConsolidateIteration: Another iteration of helper branch is not needed.

2015-09-20T07:02:51.407Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : open successful (17) size = 16912384, hd = 0. Type 8

2015-09-20T07:02:51.407Z| vcpu-0| I120: DISKLIB-LIB   : Resuming change tracking.

2015-09-20T07:02:51.414Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : closed.

2015-09-20T07:02:51.414Z| vcpu-0| A115: ConfigDB: Setting displayName = "es-1.company.local"

2015-09-20T07:02:51.422Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000001-delta.vmdk" : open successful (1041) size = 16912384, hd = 0. Type 8

2015-09-20T07:02:51.422Z| vcpu-0| I120: DISKLIB-LIB   : Resuming change tracking.

2015-09-20T07:02:51.433Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000001-delta.vmdk" : closed.

2015-09-20T07:02:51.450Z| vcpu-0| I120: SnapshotVMXConsolidateOnlineCB: nextState = 1 uid 0

2015-09-20T07:02:51.450Z| vcpu-0| I120: Closing all the disks of the VM.

2015-09-20T07:02:51.450Z| vcpu-0| I120: Closing disk scsi0:1

2015-09-20T07:02:51.453Z| vcpu-0| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 1646984.

2015-09-20T07:02:51.453Z| vcpu-0| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2015-09-20T07:02:51.483Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local_1-000001-delta.vmdk" : closed.

2015-09-20T07:02:51.483Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local_1-flat.vmdk" : closed.

2015-09-20T07:02:51.483Z| vcpu-0| I120: SNAPSHOT: SnapshotCombineDisks: Consolidating from '/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002.vmdk' to '/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local.vmdk'.

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-flat.vmdk" : open successful (24) size = 68719476736, hd = 1581449. Type 3

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-DSCPTR: Opened [0]: "es-1.company.local-flat.vmdk" (0x18)

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local.vmdk' (0x18): vmfs, 134217728 sectors / 64 GB.

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-LIB   : Resuming change tracking.

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 1581449.

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-CBT   : Successfuly created cbt node 182189-cbt.

2015-09-20T07:02:51.485Z| vcpu-0| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/182189-cbt

2015-09-20T07:02:51.486Z| vcpu-0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local.vmdk" (flags 0x18, type vmfs).

2015-09-20T07:02:51.487Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : open successful (24) size = 16912384, hd = 1843596. Type 8

2015-09-20T07:02:51.487Z| vcpu-0| I120: DISKLIB-DSCPTR: Opened [0]: "es-1.company.local-000002-delta.vmdk" (0x18)

2015-09-20T07:02:51.487Z| vcpu-0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002.vmdk' (0x18): vmfsSparse, 134217728 sectors / 64 GB.

2015-09-20T07:02:51.487Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2015-09-20T07:02:51.487Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 1843596, extentType = 0

2015-09-20T07:02:51.488Z| vcpu-0| I120: DISKLIB-LIB   : Resuming change tracking.

2015-09-20T07:02:51.488Z| vcpu-0| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 1843596.

2015-09-20T07:02:51.488Z| vcpu-0| I120: DISKLIB-CBT   : Successfuly created cbt node 1b218d-cbt.

2015-09-20T07:02:51.488Z| vcpu-0| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/1b218d-cbt

2015-09-20T07:02:51.488Z| vcpu-0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002.vmdk" (flags 0x18, type vmfsSparse).

2015-09-20T07:02:51.490Z| vcpu-0| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 1843596.

2015-09-20T07:02:51.490Z| vcpu-0| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2015-09-20T07:02:51.492Z| vcpu-0| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 1581449.

2015-09-20T07:02:51.492Z| vcpu-0| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2015-09-20T07:02:51.500Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1

2015-09-20T07:02:51.500Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 1581449, extentType = 2

2015-09-20T07:02:51.500Z| vcpu-0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 1843596, extentType = 0

2015-09-20T07:02:51.500Z| vcpu-0| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 1843596.

2015-09-20T07:02:51.500Z| vcpu-0| I120: DISKLIB-CBT   : Successfuly created cbt node 1d218d-cbt.

2015-09-20T07:02:51.500Z| vcpu-0| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/1d218d-cbt

2015-09-20T07:02:51.732Z| vcpu-0| I120: DISKLIB-LIB   : Upward Combine 2 links at 0. Need 0 MB of free space (4680409 MB available)

2015-09-20T07:02:51.736Z| vcpu-0| I120: DDB: "longContentID" = "aa8be979a63829fd10c5231db538b5bf" (was "523aed1c88135605b76574b6933eceb3")

2015-09-20T07:02:51.777Z| vcpu-0| I120: DISKLIB-CTK   : End Combine

2015-09-20T07:02:51.783Z| vcpu-0| I120: DISKLIB-CTK   : Unlinked /vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-ctk.vmdk, tmp file: /vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-ctk.vmdk-tmp

2015-09-20T07:02:51.849Z| vcpu-0| I120: DISKLIB-CTK   : resuming /vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-ctk.vmdk-tmp

2015-09-20T07:02:51.850Z| vcpu-0| I120: DISKLIB-CTK   : Renaming: /vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-ctk.vmdk-tmp -> /vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-ctk.vmdk

2015-09-20T07:02:51.851Z| vcpu-0| I120: DISKLIB-CTK   : Attempting unlink of /vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-ctk.vmdk-tmp

2015-09-20T07:02:51.853Z| vcpu-0| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 1843596.

2015-09-20T07:02:51.853Z| vcpu-0| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2015-09-20T07:02:51.861Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : closed.

2015-09-20T07:02:51.861Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-flat.vmdk" : closed.

2015-09-20T07:02:51.861Z| vcpu-0| A115: ConfigDB: Setting displayName = "es-1.company.local"

2015-09-20T07:02:51.862Z| vcpu-0| A115: ConfigDB: Setting scsi0:0.fileName = "es-1.company.local.vmdk"

2015-09-20T07:02:51.875Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : open successful (1041) size = 16912384, hd = 0. Type 8

2015-09-20T07:02:51.875Z| vcpu-0| I120: DISKLIB-LIB   : Resuming change tracking.

2015-09-20T07:02:51.885Z| vcpu-0| I120: DISKLIB-VMFS  : "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/es-1.company.local-000002-delta.vmdk" : closed.

2015-09-20T07:02:51.890Z| vcpu-0| A115: ConfigDB: Setting displayName = "es-1.company.local"

2015-09-20T07:02:51.897Z| vcpu-0| I120: SNAPSHOT: SnapshotDiskTreeFind: Detected node change from 'scsi0:0' to ''.

2015-09-20T07:02:51Z[+0.000]| vcpu-0| W110: Caught signal 11 -- tid 35868 (addr 98)

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: rip 0x18e79357 rsp 0x3fffb14f910 rbp 0x3fffb14fa00

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: rax 0x3236cc40 rbx 0x32356140 rcx 0x50 rdx 0x32356140 rsi 0x3236cc40 rdi 0x325ec4a0

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120:         r8 0x3fffb14f66b r9 0x6f72662065676e61 r10 0x0 r11 0x0 r12 0x3236cc40 r13 0x325ec4a0 r14 0x325e58b0 r15 0x0

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F910 : 0x0000000000000000 0x0000000000000010

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F920 : 0x000003fffb14f9c0 0x0000000000000000

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F930 : 0x0000000000000000 0x0000000000000000

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F940 : 0x0000000000000000 0x00000000325a9ea0

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F950 : 0x000003ff00000000 0x0000000018c66278

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F960 : 0x000003fffb14f9a8 0x000003fffb14f9d8

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F970 : 0x00000000325e58b0 0x0000000032596930

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SIGNAL: stack 3FFFB14F980 : 0x000003fffb14f9c0 0x0000000018c66534

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: Backtrace:

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: Backtrace[0] 000003fffb14f430 rip=0000000018e934fe rbx=0000000018e92cd0 rbp=000003fffb14f450 r12=0000000000000000 r13=000003fffb150680 r14=000003fffb14f990 r15=000000000000000b

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: Backtrace[1] 000003fffb14f460 rip=000000001899770c rbx=000000000000000b rbp=000003fffb14f630 r12=0000000000000003 r13=000003fffb150680 r14=000003fffb14f990 r15=000000000000000b

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: Backtrace[2] 000003fffb14f640 rip=000000000036c00f rbx=0000000032356140 rbp=000003fffb14f880 r12=000003fffb14f6c0 r13=00000000325ec4a0 r14=00000000325e58b0 r15=0000000000000000

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SymBacktrace[0] 000003fffb14f430 rip=0000000018e934fe in function (null) in object /bin/vmx loaded at 000000001873f000

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SymBacktrace[1] 000003fffb14f460 rip=000000001899770c in function (null) in object /bin/vmx loaded at 000000001873f000

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: SymBacktrace[2] 000003fffb14f640 rip=000000000036c00f

2015-09-20T07:02:51Z[+0.000]| vcpu-0| I120: Unexpected signal: 11.

2015-09-20T07:02:51Z[+3.088]| vcpu-0| W110: A core file is available in "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/vmx-zdump.000"

2015-09-20T07:02:51Z[+3.088]| vcpu-0| W110: Writing monitor corefile "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/vmmcores.gz"

2015-09-20T07:02:51Z[+3.093]| vcpu-0| I120: Counting amount of anonymous memory

2015-09-20T07:02:51Z[+3.106]| vcpu-0| I120: Total Count of Anon Pages and CR3 pages 20226

2015-09-20T07:02:51Z[+3.113]| vcpu-0| W110: Dumping core for vcpu-0

2015-09-20T07:02:51Z[+3.113]| vcpu-0| I120: CoreDump: dumping core with superuser privileges

2015-09-20T07:02:51Z[+3.113]| vcpu-0| I120: VMK Stack for vcpu 0 is at 0x4123af755000

2015-09-20T07:02:51Z[+3.113]| vcpu-0| I120: Beginning monitor coredump

2015-09-20T07:02:51Z[+3.949]| vcpu-0| I120: End monitor coredump

2015-09-20T07:02:51Z[+3.949]| vcpu-0| W110: Dumping core for vcpu-1

2015-09-20T07:02:51Z[+3.949]| vcpu-0| I120: CoreDump: dumping core with superuser privileges

2015-09-20T07:02:51Z[+3.950]| vcpu-0| I120: VMK Stack for vcpu 1 is at 0x4123afc15000

2015-09-20T07:02:51Z[+3.950]| vcpu-0| I120: Beginning monitor coredump

2015-09-20T07:02:51Z[+4.756]| vcpu-0| I120: End monitor coredump

2015-09-20T07:02:51Z[+4.756]| vcpu-0| W110: Dumping extended monitor data

2015-09-20T07:02:51Z[+9.169]| vcpu-0| I120: CoreDump: ei->size 133267456 : len = 133267456

2015-09-20T07:02:51Z[+9.172]| vcpu-0| I120: Backtrace:

2015-09-20T07:02:51Z[+9.172]| vcpu-0| I120: Backtrace[0] 000003fffb14ef30 rip=0000000018e934fe rbx=0000000018e92cd0 rbp=000003fffb14ef50 r12=0000000000000000 r13=000003fffb150680 r14=000003fffb14f990 r15=000000000000000b

2015-09-20T07:02:51Z[+9.172]| vcpu-0| I120: Backtrace[1] 000003fffb14ef60 rip=00000000188b57c5 rbx=00000000198a98a8 rbp=000003fffb14f450 r12=0000000000000001 r13=000003fffb150680 r14=000003fffb14f990 r15=000000000000000b

2015-09-20T07:02:51Z[+9.172]| vcpu-0| I120: Backtrace[2] 000003fffb14f460 rip=0000000018997766 rbx=000000000000000b rbp=000003fffb14f630 r12=0000000000000003 r13=000003fffb150680 r14=000003fffb14f990 r15=000000000000000b

2015-09-20T07:02:51Z[+9.172]| vcpu-0| I120: Backtrace[3] 000003fffb14f640 rip=000000000036c00f rbx=0000000032356140 rbp=000003fffb14f880 r12=000003fffb14f6c0 r13=00000000325ec4a0 r14=00000000325e58b0 r15=0000000000000000

2015-09-20T07:02:51Z[+9.172]| vcpu-0| I120: SymBacktrace[0] 000003fffb14ef30 rip=0000000018e934fe in function (null) in object /bin/vmx loaded at 000000001873f000

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: SymBacktrace[1] 000003fffb14ef60 rip=00000000188b57c5 in function (null) in object /bin/vmx loaded at 000000001873f000

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: SymBacktrace[2] 000003fffb14f460 rip=0000000018997766 in function (null) in object /bin/vmx loaded at 000000001873f000

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: SymBacktrace[3] 000003fffb14f640 rip=000000000036c00f

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: Msg_Post: Error

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120+ Unexpected signal: 11.

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: [msg.panic.haveLog] A log file is available in "/vmfs/volumes/526bda53-1f2b17a6-2ebf-001b21a44f80/Virtual Machines/Production/es-1.company.local/vmware.log".

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: [msg.panic.requestSupport.withoutLog] You can request support.

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: [msg.panic.requestSupport.vmSupport.vmx86]

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120+ To collect data to submit to VMware technical support, run "vm-support".

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: [msg.panic.response] We will respond on the basis of your support entitlement.

2015-09-20T07:02:51Z[+9.173]| vcpu-0| I120: ----------------------------------------

2015-09-20T07:02:51Z[+9.179]| vcpu-0| I120: Exiting

[Jeroenix]

Same issue: cluster of BL460C's on iSCSI to a 3Par7200. Backed up 213 VM's using VeeAm 8, and one of the VM's crashed and failed over while VeeAm was removing the snapshot. It happened on one of the very few VM's that had an upgraded VMtools (I hadn't come round to updating the Tools on all VMs, so only 10 of them are running upgraded versions).

Are there more people like KBAdmin and me who experience more frequent crashes with updated VMtools than VMs with old VMtools?

[JumpMaster]

I have 23 ESXi hosts upgraded to Update 3 and am seeing this issue with veeam.  This started happening shortly after upgrading to veeam 8 so had tickets open with veeam and vmware.  After two days of sending log, after log, after log, (you know the drill) they told us about this issue.  All the time we were continuing our upgrade to update 3.

It didn't sound like they were close to a fix.

[fr8rt8rt]

what they told you about this issue?i want to upgrade to u3!

[JumpMaster]

DON'T!

[MikeStone226]

I started reverting my hosts back to U2 (VMware KB: Reverting to a previous version of ESXi).  since I haven't received *ANYTHING* on my Support Request (which was shocking, usually support is great). I rolled back a host for testing and found that the guests are now only visible through the vSphere Web Client and not the vSphere Client.  That might be specific to me, but who knows, just a heads up.

[KBadmin]

@ MikeStone226

Had the same Problem after reverting my Hosts. I think you must connect every VM on the vSphere Client manually too.

Hope vmware fix this bug soon!

[igonzalez82]

Same problem here.

I've some ESXi in version 5.5 Update 2, and the issue doesn't happen.

I'm also using Veeam and multivendor 10GB SANs (Netapp + Solidfire)

Someone have more information from VMware?

Do you recommend to downgrade, disabling CBT can cause problems if you SAN infrastructure is a bit busy.

[airfrog7]

I have a case raised with VMware about this issue. The tech support guy I spoke to acknowledged this is a pretty major bug. Something is causing the Windows guest (I don't know if it affects Linux) to crash on snapshot removal. They don't have a workaround other than downgrading, which we are currently doing. This issue has apparently been escalated to the highest level as it is affecting an awful lot of customers. They don't have an ETA for when a fix will be available.

The error you see in the vmware.log file for the VM will look something like this:

2015-09-29T17:24:42Z[+8.996]| vcpu-0| I120: SymBacktrace[1] 000003fffbf1af60 rip=00000000162957c5 in function (null) in object /bin/vmx loaded at 000000001611f000

2015-09-29T17:24:42Z[+8.996]| vcpu-0| I120: SymBacktrace[2] 000003fffbf1b460 rip=0000000016377766 in function (null) in object /bin/vmx loaded at 000000001611f000

2015-09-29T17:24:42Z[+8.996]| vcpu-0| I120: SymBacktrace[3] 000003fffbf1b640 rip=00000000003d500f

2015-09-29T17:24:42Z[+8.996]| vcpu-0| I120: Msg_Post: Error

2015-09-29T17:24:42Z[+8.996]| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)

2015-09-29T17:24:42Z[+8.996]| vcpu-0| I120+ Unexpected signal: 11.

In our case it affecting maybe 1% of our VMs every evening, but different VMs each time.

[KBadmin]

I think so, it´s causing Windows guests.

We had this Problems only on 64 bit Systems of Server´s 2008 und Server´s 2012 R2. We have 2 VM´s (Windows Server 2003 - 32bit) we need them for old applications - this Systems never crashed!

Our Linux Server´s never crashed, too - because on the Linux machines are other vmtools.

We had the same effect, different machines crashed after deleting snapshot´s.

After rollback our hosts to update 2 we haven´t Problems.

[Sergey_Petrushi]

Not only Windows guests. We have same issue with Debian based VM's.

[vAMenezes]

Is everyone here using Veem? Or is this happening with different backup tools? Anybody here using Commvault? I have upgraded to U3 but these are new hosts so I don't have anything in production running on them yet, so if I'm going to downgrade I need to do it now.

[gregsn]

I've been able to reproduce the problem by manually creating and deleting snapshots so I don't think it's related to any particular backup software.

[drc0106]

I have been working on a simular issue with VMWare when Update 2 came out. It was in relation to snapshots with quiesced turned on. Check out form Windows 7 and Windows2008R2 VM BSOD ntfs.sys and KB2115997.

We have not installed Update 3 yet to fix the issue so I have emailed the engineer I have been working with and seeing if he has any insight into the issue. You may want to see if turning off quiesce snapshots and seeing if that works. If so, reinstall the VMWare Tools without the VSS Writer. The issue we are having is related to the fact that Update 2 changed the VSS Writer and was causing our Windows servers to BSOD when snapshots were created. They were suppose to roll back to the old VSS Writer in Update 3 to fix this issue. That rollback may have caused some issues. However, I am only speculating here as we have not updated to Update 3 yet or experienced the issue when snapshots are deleted only when created.

[Jeroenix]

Because in my case, very few VMs go down during backup (only one, to be precise) I decided to let the VeeAm backups run. Last night, guess what: the very same VM went down. Out of 213 VMs. I checked out this one VM but can't find anything out of the ordinary. I also opened a ticket with VMware, maybe if they collect all our logs, they can find a pattern.

[igonzalez82]

Hi,

News from the support team:

"Last night engineering found the root cause. They will need to produce an express patch.

We have asked how long are we expecting to wait for this, I will follow up once a response has been received."

[Fantomas01]

Would this bug be causing issues with Server 2012 VM's not booting up.  We havent deleted any snapshots.  I patched our hosts yesterday as we were having an issue with a couple of our 2012 VMs getting stuck on the splash screen and not progressing.

The KB that I read about that issue said the issue was fixed in u3.

Thanks

[GMZSE]

Still shocked there is no mention of this major issue when you are at the download page. Does anyone know an ETA for the fix?

[mfedermanv]

Does this issue still exist for you, and what version of NetBackup?

[suprnova13]

I have this same issue.  Spoke with support, here is the KB article: VMware KB: Snapshot consolidation causes virtual machines running on VMware ESXi 5.5 Update 3 hosts ...

He did say it should be fixed in express patch 8, but that won't be released for another two to three weeks.

[admin]

The following KB article has been updated with additional workarounds as well as the symptoms cleaned up:

Hope this helps!

http://kb.vmware.com/kb/2133118

Snapshot consolidation causes virtual machines running on VMware ESXi 5.5 Update 3 hosts to fail with the error: Unexpected signal: 11 (2133118)