Linux VM won't come up - Remove console on <server> disconnected

[heldstma]

We are having an issue where a Suse Linux VM won't come up. It went down to zero CPU and I couldn't even power it down through the VM.

We just shut it down using the STOP button in VC, and it said that it was

Unable to communicate with the remote host, since it is disconnected.

Powering it back on just hangs at 95%.

Anyone have a thought or starting point?

Matt

[waynegrow]

Have you looked at the vmware.log for the VM?

[acmcnick]

What kind of Disk/Protocol do your vm's reside on?

[heldstma]

Anything here that looks odd? The server crashed at 14:11 on 7/17

Jul 11 17:21:58.761: vmx| Log for VMware ESX Server pid=101075 version=3.0.1 build=build-32039 option=Release

Jul 11 17:21:58.773: vmx| Hostname=wfpesx2.mnscu.edu

Jul 11 17:21:58.773: vmx| Command line: "/usr/lib/vmware/bin/vmware-vmx" "-ssched.group=host/user" "-@" "pipe=/tmp/vmhsdaemon-0/vmx9f10e0906ef5b20e;vm=9f10e0906ef5b20e" "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmx"

Jul 11 17:21:58.832: vmx| UI Connecting to pipe '/tmp/vmhsdaemon-0/vmx9f10e0906ef5b20e' with user '(null)'

Jul 11 17:21:58.959: vmx| pcpu #0 CPUID numEntries=5 GenuntelineI

Jul 11 17:21:58.960: vmx| pcpu #0 CPUID version=0xf4a id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x20800

Jul 11 17:21:58.960: vmx| pcpu #0 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x1

Jul 11 17:21:58.961: vmx| pcpu #1 CPUID numEntries=5 GenuntelineI

Jul 11 17:21:58.961: vmx| pcpu #1 CPUID version=0xf4a id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x1020800

Jul 11 17:21:58.961: vmx| pcpu #1 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x1

Jul 11 17:21:58.961: vmx| pcpu #2 CPUID numEntries=5 GenuntelineI

Jul 11 17:21:58.961: vmx| pcpu #2 CPUID version=0xf4a id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x6020800

Jul 11 17:21:58.961: vmx| pcpu #2 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x1

Jul 11 17:21:58.961: vmx| pcpu #3 CPUID numEntries=5 GenuntelineI

Jul 11 17:21:58.961: vmx| pcpu #3 CPUID version=0xf4a id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x7020800

Jul 11 17:21:58.961: vmx| pcpu #3 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x1

Jul 11 17:21:58.961: vmx| CPUID id1.edx: 0xbfebfbff id1.ecx: 0x659d id81.edx: 0x20000000 id81.ecx: 0x1

Jul 11 17:21:58.961: vmx| CPUID id88.ecx: 0 id88.edx: 0

Jul 11 17:21:58.961: vmx| CPUID Maximum Physical Address Bits supported across all CPUs : 36

Jul 11 17:21:59.112: vmx| Setup symlink /var/run/vmware/%2fvmfs%2fvolumes%2f4610f74d%2d7d273bb1%2d47fe%2d0016356cc665%2fgwwebdom2%2fgwwebdom2%2evmx -> /var/run/vmware/root_0/1184192518782116_101075

Jul 11 17:21:59.113: vmx| ACL_InitCapabilities: current IPC thread

Jul 11 17:21:59.114: vmx| ACL_InitCapabilities: done

Jul 11 17:21:59.114: vmx| changing directory to /vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/.

Jul 11 17:21:59.114: vmx| Config file: /vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmx

Jul 11 17:21:59.189: vmx| DISKLIB-VMFS : "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2-flat.vmdk" : open successful (23) size = 26843545600, hd = -1. Type 3

Jul 11 17:21:59.190: vmx| DISKLIB-VMFS : "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2-flat.vmdk" : closed.

Jul 11 17:21:59.193: vmx| DISKLIB-VMFS : "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1-flat.vmdk" : open successful (23) size = 16106127360, hd = -1. Type 3

Jul 11 17:21:59.195: vmx| DISKLIB-VMFS : "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1-flat.vmdk" : closed.

Jul 11 17:21:59.369: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Jul 11 17:21:59.793: vmx| VMXVmdbCbVmVmxExecState: Exec state change requested to state poweredOn without reset

Jul 11 17:21:59.794: vmx| TOOLS delaying state change request to state 3

Jul 11 17:21:59.794: vmx| PowerOn

Jul 11 17:21:59.801: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Jul 11 17:22:00.097: vmx| VMMon_CreateVM: vmmon.numVCPUs=1

Jul 11 17:22:00.123: vmx| UNAME Linux wfpesx2.mnscu.edu 2.4.21-37.0.2.ELvmnix #1 Mon Sep 25 22:18:34 PDT 2006 i686 (uwglibc version 4)

Jul 11 17:22:00.123: vmx| DICT --- USER PREFERENCES

Jul 11 17:22:00.123: vmx| DICT --- USER DEFAULTS

Jul 11 17:22:00.123: vmx| DICT --- HOST DEFAULTS

Jul 11 17:22:00.123: vmx| DICT libdir = /usr/lib/vmware

Jul 11 17:22:00.123: vmx| DICT authd.fullpath = /usr/sbin/vmware-authd

Jul 11 17:22:00.123: vmx| DICT authd.client.port = 902

Jul 11 17:22:00.123: vmx| DICT vmware.fullpath = /usr/bin/vmware

Jul 11 17:22:00.123: vmx| DICT control.fullpath = /usr/bin/vmware-cmd

Jul 11 17:22:00.123: vmx| DICT serverd.fullpath = /usr/sbin/vmware-serverd

Jul 11 17:22:00.123: vmx| DICT serverd.init.fullpath = /usr/lib/vmware/serverd/init.pl

Jul 11 17:22:00.123: vmx| DICT authd.proxy.vim = vmware-hostd:hostd-vmdb

Jul 11 17:22:00.123: vmx| DICT authd.proxy.nfc = vmware-hostd:ha-nfc

Jul 11 17:22:00.123: vmx| DICT --- SITE DEFAULTS

Jul 11 17:22:00.123: vmx| DICT tag.help = introduction.htm

Jul 11 17:22:00.123: vmx| DICT tag.configurationEditor = config_editor_newvm.htm

Jul 11 17:22:00.123: vmx| DICT tag.ideConfig = devices_virtualdrive.htm

Jul 11 17:22:00.123: vmx| DICT tag.floppyConfig = devices_floppy.htm

Jul 11 17:22:00.123: vmx| DICT tag.mouseConfig = devices_mouse.htm

Jul 11 17:22:00.123: vmx| DICT tag.netConfig = devices_netadapter.htm

Jul 11 17:22:00.123: vmx| DICT tag.parallelConfig = devices_parallel.htm

Jul 11 17:22:00.123: vmx| DICT tag.serialConfig = devices_serial.htm

Jul 11 17:22:00.123: vmx| DICT tag.soundConfig = devices_sound.htm

Jul 11 17:22:00.123: vmx| DICT tag.memConfig = configvm_memory.htm

Jul 11 17:22:00.123: vmx| DICT tag.miscConfig = configvm.htm

Jul 11 17:22:00.123: vmx| DICT tag.usbConfig = devices_usb.htm

Jul 11 17:22:00.123: vmx| DICT tag.displayConfig = configvm_display-problems.htm

Jul 11 17:22:00.123: vmx| DICT tag.tools = vmtools.htm

Jul 11 17:22:00.123: vmx| DICT --- COMMAND LINE

Jul 11 17:22:00.123: vmx| DICT sched.group = host/user

Jul 11 17:22:00.123: vmx| DICT --- CONFIGURATION

Jul 11 17:22:00.123: vmx| DICT config.version = 8

Jul 11 17:22:00.123: vmx| DICT virtualHW.version = 4

Jul 11 17:22:00.123: vmx| DICT floppy0.present = TRUE

Jul 11 17:22:00.123: vmx| DICT nvram = gwwebdom2.nvram

Jul 11 17:22:00.123: vmx| DICT powerType.powerOff = default

Jul 11 17:22:00.123: vmx| DICT powerType.powerOn = default

Jul 11 17:22:00.123: vmx| DICT powerType.suspend = default

Jul 11 17:22:00.123: vmx| DICT powerType.reset = default

Jul 11 17:22:00.123: vmx| DICT displayName = gwwebdom

Jul 11 17:22:00.123: vmx| DICT extendedConfigFile = gwwebdom2.vmxf

Jul 11 17:22:00.123: vmx| DICT scsi0.present = TRUE

Jul 11 17:22:00.123: vmx| DICT scsi0.sharedBus = none

Jul 11 17:22:00.123: vmx| DICT scsi0.virtualDev = lsilogic

Jul 11 17:22:00.123: vmx| DICT memsize = 1536

Jul 11 17:22:00.123: vmx| DICT scsi0:0.present = TRUE

Jul 11 17:22:00.123: vmx| DICT scsi0:0.fileName = gwwebdom2.vmdk

Jul 11 17:22:00.123: vmx| DICT scsi0:0.deviceType = scsi-hardDisk

Jul 11 17:22:00.123: vmx| DICT scsi0:1.present = TRUE

Jul 11 17:22:00.123: vmx| DICT scsi0:1.fileName = gwwebdom2_1.vmdk

Jul 11 17:22:00.123: vmx| DICT scsi0:1.deviceType = scsi-hardDisk

Jul 11 17:22:00.123: vmx| DICT ide0:0.present = TRUE

Jul 11 17:22:00.123: vmx| DICT ide0:0.clientDevice = TRUE

Jul 11 17:22:00.124: vmx| DICT ide0:0.deviceType = atapi-cdrom

Jul 11 17:22:00.124: vmx| DICT ide0:0.startConnected = FALSE

Jul 11 17:22:00.124: vmx| DICT floppy0.startConnected = FALSE

Jul 11 17:22:00.124: vmx| DICT floppy0.fileName = /dev/fd0

Jul 11 17:22:00.124: vmx| DICT floppy0.clientDevice = TRUE

Jul 11 17:22:00.124: vmx| DICT ethernet0.present = TRUE

Jul 11 17:22:00.124: vmx| DICT ethernet0.allowGuestConnectionControl = FALSE

Jul 11 17:22:00.124: vmx| DICT ethernet0.networkName = VM Network

Jul 11 17:22:00.124: vmx| DICT ethernet0.addressType = vpx

Jul 11 17:22:00.124: vmx| DICT ethernet0.generatedAddress = 00:50:56:8e:3e:5a

Jul 11 17:22:00.124: vmx| DICT guestOS = sles

Jul 11 17:22:00.124: vmx| DICT uuid.bios = 50 0e 4d 5e 5c 6f 65 a0-b2 2c d2 cd 05 3b 99 9d

Jul 11 17:22:00.124: vmx| DICT log.fileName = vmware.log

Jul 11 17:22:00.124: vmx| DICT sched.cpu.min = 0

Jul 11 17:22:00.124: vmx| DICT sched.cpu.units = mhz

Jul 11 17:22:00.124: vmx| DICT sched.cpu.shares = normal

Jul 11 17:22:00.124: vmx| DICT sched.mem.minsize = 0

Jul 11 17:22:00.124: vmx| DICT sched.mem.shares = normal

Jul 11 17:22:00.124: vmx| DICT toolScripts.afterPowerOn = TRUE

Jul 11 17:22:00.124: vmx| DICT toolScripts.afterResume = TRUE

Jul 11 17:22:00.124: vmx| DICT toolScripts.beforeSuspend = TRUE

Jul 11 17:22:00.124: vmx| DICT toolScripts.beforePowerOff = TRUE

Jul 11 17:22:00.124: vmx| DICT scsi0:0.redo =

Jul 11 17:22:00.124: vmx| DICT scsi0:1.redo =

Jul 11 17:22:00.124: vmx| DICT tools.syncTime = FALSE

Jul 11 17:22:00.124: vmx| DICT vmware.tools.requiredversion = 7201

Jul 11 17:22:00.124: vmx| DICT uuid.location = 56 4d da dd b8 68 1e 26-78 84 17 0f f4 77 a0 87

Jul 11 17:22:00.124: vmx| DICT migrate.hostlog = ./gwwebdom2-e08f11eb.hlog

Jul 11 17:22:00.124: vmx| DICT sched.swap.derivedName = /vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2-e08f11eb.vswp

Jul 11 17:22:00.124: vmx| DICT --- USER DEFAULTS

Jul 11 17:22:00.124: vmx| DICT --- HOST DEFAULTS

Jul 11 17:22:00.124: vmx| DICT libdir = /usr/lib/vmware

Jul 11 17:22:00.124: vmx| DICT authd.fullpath = /usr/sbin/vmware-authd

Jul 11 17:22:00.124: vmx| DICT authd.client.port = 902

Jul 11 17:22:00.124: vmx| DICT vmware.fullpath = /usr/bin/vmware

Jul 11 17:22:00.124: vmx| DICT control.fullpath = /usr/bin/vmware-cmd

Jul 11 17:22:00.124: vmx| DICT serverd.fullpath = /usr/sbin/vmware-serverd

Jul 11 17:22:00.124: vmx| DICT serverd.init.fullpath = /usr/lib/vmware/serverd/init.pl

Jul 11 17:22:00.124: vmx| DICT authd.proxy.vim = vmware-hostd:hostd-vmdb

Jul 11 17:22:00.124: vmx| DICT authd.proxy.nfc = vmware-hostd:ha-nfc

Jul 11 17:22:00.124: vmx| DICT --- SITE DEFAULTS

Jul 11 17:22:00.124: vmx| DICT tag.help = introduction.htm

Jul 11 17:22:00.124: vmx| DICT tag.configurationEditor = config_editor_newvm.htm

Jul 11 17:22:00.124: vmx| DICT tag.ideConfig = devices_virtualdrive.htm

Jul 11 17:22:00.124: vmx| DICT tag.floppyConfig = devices_floppy.htm

Jul 11 17:22:00.124: vmx| DICT tag.mouseConfig = devices_mouse.htm

Jul 11 17:22:00.124: vmx| DICT tag.netConfig = devices_netadapter.htm

Jul 11 17:22:00.124: vmx| DICT tag.parallelConfig = devices_parallel.htm

Jul 11 17:22:00.124: vmx| DICT tag.serialConfig = devices_serial.htm

Jul 11 17:22:00.124: vmx| DICT tag.soundConfig = devices_sound.htm

Jul 11 17:22:00.124: vmx| DICT tag.memConfig = configvm_memory.htm

Jul 11 17:22:00.124: vmx| DICT tag.miscConfig = configvm.htm

Jul 11 17:22:00.124: vmx| DICT tag.usbConfig = devices_usb.htm

Jul 11 17:22:00.124: vmx| DICT tag.displayConfig = configvm_display-problems.htm

Jul 11 17:22:00.124: vmx| DICT tag.tools = vmtools.htm

Jul 11 17:22:00.124: vmx| DICT --- GLOBAL SETTINGS

Jul 11 17:22:00.129: vmx| STATDECLGROUP stats Root "" null

Jul 11 17:22:00.138: vmx| Host CPUID features: version 0xf48 id1.edx 0xbfebfbff id1.ecx 0x659d id81.edx 0x20000000 id81.ecx 0x1

Jul 11 17:22:00.138: vmx| CPU.cpuFeatures = 0x39b1ff30

Jul 11 17:22:00.139: vmx| CPUID after masking: version 0xf48 id1.edx 0xfebfbff id1.ecx 0x11 id81.edx 0x800 id81.ecx 0x0 id88.ecx 0x0

Jul 11 17:22:00.139: vmx| CPU.cpuFeatures = 0x18b1fe00

Jul 11 17:22:00.294: vmx| KHZEstimate 3600128

Jul 11 17:22:00.294: vmx| MHZEstimate 3600

Jul 11 17:22:00.294: vmx| NumVCPUs 1

Jul 11 17:22:00.295: vmx| UUID: location-UUID is 56 4d da dd b8 68 1e 26-78 84 17 0f f4 77 a0 87

Jul 11 17:22:00.339: vmx| Mapped mainmem as pageable

Jul 11 17:22:00.346: vmx| MStat: Creating Stat vm.uptime

Jul 11 17:22:00.346: vmx| MStat: Creating Stat vm.suspendTime

Jul 11 17:22:00.346: vmx| MStat: Creating Stat vm.powerOnTimeStamp

Jul 11 17:22:00.439: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Jul 11 17:22:00.450: vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmdk' persistent R\[(null)]

Jul 11 17:22:00.485: vmx| DISKLIB-VMFS : "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2-flat.vmdk" : open successful (10) size = 26843545600, hd = 1048. Type 3

Jul 11 17:22:00.485: vmx| DISKLIB-DSCPTR: Opened : "gwwebdom2-flat.vmdk" (0xa)

Jul 11 17:22:00.486: vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmdk' (0xa): vmfs, 52428800 sectors / 25600 Mb.

Jul 11 17:22:00.488: vmx| DISKLIB-LIB : Opened "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmdk" (flags 0xa). 0x856f1c0

Jul 11 17:22:00.542: vmx| DISK: OPEN '/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmdk' Geo (3263/255/63) BIOS Geo (0/0/0) freeSpace=580520Mb, DDB.adapterType=lsilogic

Jul 11 17:22:00.566: vmx| DISK: OPEN scsi0:1 '/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1.vmdk' persistent R\[(null)]

Jul 11 17:22:00.582: vmx| DISKLIB-VMFS : "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1-flat.vmdk" : open successful (10) size = 16106127360, hd = 1049. Type 3

Jul 11 17:22:00.582: vmx| DISKLIB-DSCPTR: Opened : "gwwebdom2_1-flat.vmdk" (0xa)

Jul 11 17:22:00.582: vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1.vmdk' (0xa): vmfs, 31457280 sectors / 15360 Mb.

Jul 11 17:22:00.583: vmx| DISKLIB-LIB : Opened "/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1.vmdk" (flags 0xa). 0x856b8c8

Jul 11 17:22:00.640: vmx| DISK: OPEN '/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1.vmdk' Geo (1958/255/63) BIOS Geo (0/0/0) freeSpace=580520Mb, DDB.adapterType=lsilogic

Jul 11 17:22:00.657: vmx| TimeTracker host to guest rate conversion 1327772671416 @ 3600128000Hz -> 1327772671416 @ 3600128000Hz

Jul 11 17:22:00.657: vmx| TimeTracker host to guest rate conversion ((x * 2147483648) >> 31) + 0

Jul 11 17:22:00.688: vmx| DumpDiskInfo: scsi0:0 createType=11, capacity = 52428800, numLinks = 1, allocationType = 0

Jul 11 17:22:00.688: vmx| SCSIDiskESXPopulateVDevDesc: Using FS backend

Jul 11 17:22:00.688: vmx| DumpDiskInfo: scsi0:1 createType=11, capacity = 31457280, numLinks = 1, allocationType = 0

Jul 11 17:22:00.688: vmx| SCSIDiskESXPopulateVDevDesc: Using FS backend

Jul 11 17:22:00.689: vmx| SCSI0: UNTAGGED commands will be converted to ORDER tags.

Jul 11 17:22:00.689: vmx| MStat: Creating Stat vm.heartbeat

Jul 11 17:22:00.690: vmx| DISKUTIL: scsi0:1 : toolsVersion = 0

Jul 11 17:22:00.690: vmx| DISKUTIL: scsi0:0 : toolsVersion = 0

Jul 11 17:22:00.690: vmx| DISKUTIL: Offline toolsVersion = 0

Jul 11 17:22:00.691: vmx| TOOLS INSTALL initializing state to IDLE on power on.

Jul 11 17:22:00.718: vmx| DVGA: Cannot map VGA physical frame buffer: Invalid argument.

Jul 11 17:22:00.718: vmx| DVGA: Full screen VGA will not be available.

Jul 11 17:22:00.979: vmx| SVGA: Truncated max res to VRAM size: 4194304 bytes VRAM, 1180x885 Max Resolution

Jul 11 17:22:00.979: vmx| VLANCE: send cluster threshold is 80, size = 2 recalcInterval is 2 ticks

Jul 11 17:22:00.980: vmx| Ethernet0 MAC Address: 00:50:56:8e:3e:5a

Jul 11 17:22:01.044: vmx| VMX_PowerOn: ModuleTable_PowerOn = 1

Jul 11 17:22:01.044: vmx| VMX setting maximum IPC write buffers to 0 packets, 0 bytes

Jul 11 17:22:01.049: mks| Async MKS thread is alive

Jul 11 17:22:01.056: vcpu-0| VMMon_Start: vcpu-0: worldID=1079

Jul 11 17:22:01.057: vcpu-0| APIC: version = 0x14, max LVT = 5

Jul 11 17:22:01.057: vcpu-0| APIC: LDR = 0x4000000, DFR = 0xffffffff

Jul 11 17:22:01.103: vcpu-0| guestCpuFeatures = 0x18b1fe00

Jul 11 17:22:01.103: vcpu-0| Init modules.

Jul 11 17:22:01.111: vcpu-0| CPU reset: hard

Jul 11 17:22:01.120: vcpu-0| sz=3035104

Jul 11 17:22:01.301: vcpu-0| vmm32 initialized: Releasebuild-32039. cflags: 0x20000002.0e304000.00000a80

Jul 11 17:22:01.729: vcpu-0| SVGA: Registering MemSpace at 0xf8000000(0x0) and 0xf4000000(0x0)

Jul 11 17:22:01.737: vcpu-0| SVGA: Unregistering MemSpace at 0xf8000000(0xf8000000) and 0xf4000000(0xf4000000)

Jul 11 17:22:01.755: vcpu-0| SVGA: Registering MemSpace at 0xf8000000(0xf8000000) and 0xf4000000(0xf4000000)

Jul 11 17:22:01.766: vcpu-0| SVGA: Unregistering MemSpace at 0xf8000000(0xf8000000) and 0xf4000000(0xf4000000)

Jul 11 17:22:01.767: vcpu-0| SVGA: Registering IOSpace at 0x1060 (0x0)

Jul 11 17:22:01.767: vcpu-0| SVGA: Registering MemSpace at 0xf8000000(0xf8000000) and 0xf4000000(0xf4000000)

Jul 11 17:22:04.707: vcpu-0| DISKUTIL: scsi0:0 : geometry=3263/255/63

Jul 11 17:22:04.707: vcpu-0| DISKUTIL: scsi0:1 : geometry=1958/255/63

Jul 11 17:22:06.647: vcpu-0| BIOS-UUID is 50 0e 4d 5e 5c 6f 65 a0-b2 2c d2 cd 05 3b 99 9d

Jul 11 17:22:07.414: vcpu-0| DISKUTIL: scsi0:1 : toolsVersion = 0

Jul 11 17:22:07.414: vcpu-0| DISKUTIL: scsi0:0 : toolsVersion = 0

Jul 11 17:22:07.415: vcpu-0| DISKUTIL: Offline toolsVersion = 0

Jul 11 17:22:16.381: mks| HostOps hideCursor before defineCursor!

Jul 11 17:22:16.676: vcpu-0| Unknown int 10h func 0x0000

Jul 11 17:22:17.820: vcpu-0| SVGA: Unregistering IOSpace at 0x1060 (0x1060)

Jul 11 17:22:17.820: vcpu-0| SVGA: Registering IOSpace at 0xfff0 (0x0)

Jul 11 17:22:17.820: vcpu-0| SVGA: Unregistering IOSpace at 0xfff0 (0xfff0)

Jul 11 17:22:17.820: vcpu-0| SVGA: Registering IOSpace at 0x1060 (0x0)

Jul 11 17:22:47.875: vcpu-0| HBACommon: First write on scsi0:0.fileName/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2.vmdk

Jul 11 17:22:48.029: vcpu-0| DISKLIB-CHAIN : UpdateContentID: old = 0xe2f66b1b, new = 0xc75fc052

Jul 11 17:23:00.864: vcpu-0| HBACommon: First write on scsi0:1.fileName/vmfs/volumes/4610f74d-7d273bb1-47fe-0016356cc665/gwwebdom2/gwwebdom2_1.vmdk

Jul 11 17:23:00.909: vcpu-0| DISKLIB-CHAIN : UpdateContentID: old = 0xc0bc61f0, new = 0xa692ca41

Jul 11 17:23:39.641: vcpu-0| CDROM: Emulate GET CONFIGURATION RT 0 start feature 40

Jul 11 17:23:39.641: vcpu-0| CDROM: Mode Sense for Unsupported Page 0x3

Jul 11 17:23:39.642: vcpu-0| CDROM: Mode Sense for Unsupported Page 0x2C

Jul 11 17:23:39.642: vcpu-0| CDROM: Emulate GET CONFIGURATION RT 0 start feature 32

Jul 11 17:24:00.564: vcpu-0| SVGA: Unregistering IOSpace at 0x1060 (0x1060)

Jul 11 17:24:00.564: vcpu-0| SVGA: Unregistering MemSpace at 0xf8000000(0xf8000000) and 0xf4000000(0xf4000000)

Jul 11 17:24:00.592: vcpu-0| SVGA: Registering IOSpace at 0x1060 (0x0)

Jul 11 17:24:00.592: vcpu-0| SVGA: Registering MemSpace at 0xf8000000(0xf8000000) and 0xf4000000(0xf4000000)

Jul 11 17:24:01.310: mks| SVGA: Using non-extended FIFO (min is 16, need 36)

Jul 11 17:24:01.365: mks| VNCENCODE 2 encoding mode change: (1024x768x16depth,16bpp)

Jul 11 17:24:01.600: vcpu-0| Guest OS = 0x5008

Jul 11 17:24:01.600: mks| HostOps hideCursor before defineCursor!

Jul 11 17:24:01.632: mks| VNCENCODE 2 encoding mode change: (1024x768x16depth,16bpp)

Jul 11 17:24:01.867: vcpu-0| VMMouse: CMD Read ID

Jul 11 17:24:01.868: vcpu-0| VMMouse: CMD Disable

Jul 11 17:24:01.868: vcpu-0| VMMouse: Disabling VMMouse mode

Jul 11 17:24:04.575: vcpu-0| VMMouse: CMD Read ID

Jul 11 17:24:04.575: vcpu-0| MKS switching absolute mouse on

Jul 11 17:31:33.592: mks| SOCKET 2 recv error 5: Input/output error

Jul 11 17:31:33.592: mks| SOCKET 2 destroying VNC backend on socket error: 5

Jul 11 17:32:00.691: vmx| TOOLS setting the tools version to '0'

Jul 12 11:46:11.861: mks| SOCKET 3 recv error 5: Input/output error

Jul 12 11:46:11.868: mks| SOCKET 3 destroying VNC backend on socket error: 5

Jul 12 16:40:10.106: mks| SOCKET 4 recv error 5: Input/output error

Jul 12 16:40:10.106: mks| SOCKET 4 destroying VNC backend on socket error: 5

Jul 12 16:50:00.108: mks| SOCKET 5 recv error 5: Input/output error

Jul 12 16:50:00.109: mks| SOCKET 5 destroying VNC backend on socket error: 5

Jul 12 17:10:32.260: mks| SOCKET 6 recv error 5: Input/output error

Jul 12 17:10:32.260: mks| SOCKET 6 destroying VNC backend on socket error: 5

Jul 13 09:34:33.299: mks| SOCKET 7 recv error 5: Input/output error

Jul 13 09:34:33.302: mks| SOCKET 7 destroying VNC backend on socket error: 5

Jul 13 13:19:26.135: mks| SOCKET 9 recv error 5: Input/output error

Jul 13 13:19:26.136: mks| SOCKET 9 destroying VNC backend on socket error: 5

Jul 13 15:36:58.781: mks| SOCKET 10 recv error 5: Input/output error

Jul 13 15:36:58.782: mks| SOCKET 10 destroying VNC backend on socket error: 5

Jul 13 16:27:53.417: mks| SOCKET 12 recv error 5: Input/output error

Jul 13 16:27:53.417: mks| SOCKET 12 destroying VNC backend on socket error: 5

Jul 13 16:32:01.816: mks| SOCKET 8 recv error 5: Input/output error

Jul 13 16:32:01.817: mks| SOCKET 8 destroying VNC backend on socket error: 5

Jul 13 16:44:07.748: mks| SOCKET 11 recv error 5: Input/output error

Jul 13 16:44:07.748: mks| SOCKET 11 destroying VNC backend on socket error: 5

Jul 16 08:40:50.518: mks| SOCKET 13 recv error 5: Input/output error

Jul 16 08:40:50.519: mks| SOCKET 13 destroying VNC backend on socket error: 5

Jul 16 12:27:43.051: mks| SOCKET 15 recv error 5: Input/output error

Jul 16 12:27:43.051: mks| SOCKET 15 destroying VNC backend on socket error: 5

Jul 16 13:27:23.767: mks| SOCKET 17 recv error 5: Input/output error

Jul 16 13:27:23.767: mks| SOCKET 17 destroying VNC backend on socket error: 5

Jul 16 14:49:05.801: mks| SOCKET 18 recv error 5: Input/output error

Jul 16 14:49:05.801: mks| SOCKET 18 destroying VNC backend on socket error: 5

Jul 16 15:19:00.379: mks| SOCKET 16 send error 32: Broken pipe

Jul 16 15:19:00.379: mks| SOCKET 16 destroying VNC backend on socket error: 32

Jul 16 15:19:00.518: mks| POLL device deleted

Jul 16 15:19:00.542: mks| SOCKET 19 send error 32: Broken pipe

Jul 16 15:19:00.542: mks| SOCKET 19 destroying VNC backend on socket error: 32

Jul 16 15:19:00.545: mks| POLL device deleted

Jul 16 15:19:24.761: mks| SOCKET 14 recv error 5: Input/output error

Jul 16 15:19:24.762: mks| SOCKET 14 destroying VNC backend on socket error: 5

Jul 16 15:45:05.648: mks| SOCKET 20 recv error 5: Input/output error

Jul 16 15:45:05.648: mks| SOCKET 20 destroying VNC backend on socket error: 5

Jul 17 07:20:10.770: mks| SOCKET 21 recv error 5: Input/output error

Jul 17 07:20:10.770: mks| SOCKET 21 destroying VNC backend on socket error: 5

Jul 17 11:36:14.015: mks| SOCKET 22 send error 32: Broken pipe

Jul 17 11:36:14.016: mks| SOCKET 22 destroying VNC backend on socket error: 32

Jul 17 11:36:14.016: mks| POLL device deleted

Jul 17 11:36:14.050: mks| SOCKET 24 send error 32: Broken pipe

Jul 17 11:36:14.052: mks| SOCKET 24 destroying VNC backend on socket error: 32

Jul 17 11:36:14.195: mks| POLL device deleted

Jul 17 12:42:49.258: mks| SOCKET 23 recv error 5: Input/output error

Jul 17 12:42:49.259: mks| SOCKET 23 destroying VNC backend on socket error: 5

Jul 17 13:54:47.426: mks| SOCKET 28 recv error 5: Input/output error

Jul 17 13:54:47.426: mks| SOCKET 28 destroying VNC backend on socket error: 5

Jul 17 14:18:06.382: mks| SOCKET 25 send error 32: Broken pipe

Jul 17 14:18:06.382: mks| SOCKET 25 destroying VNC backend on socket error: 32

Jul 17 14:18:06.383: mks| POLL device deleted

[heldstma]

Our disks are located on a SAN. The entire VMs reside there. Other VMs in the same datastore are ok.

[heldstma]

Here is what we have tried:

o Login to the service console

o Checked the VM state by typing “vmware-cmd //server.vmx start” to power on VM

We powered it on this way and it didn't come up.

I tried to clone the VM in question, and it wouldn't let me after it gave me an error of:

A file was not found. \[wfpsan2]gwwebdom2/gwwebdom2.vmdk

The file does exist though.

[acmcnick]

A couple of things I would try:

1. Verify Permissions on all the Files and the directory are correct.

it should look like this

drwxr-xr-x root root .

d----

---

root root ..

-rw------- root root test-flat.vmdk

-rw------- root root test.vmdk

-rw------- root root test.vmsd

-rwxr-xr-x root root test.vmx

-rw------- root root test.vmxf

2. Verify that data exists in the VMDK files, I have seen the Descriptor file become corrupt before.

3. Verify that data exists in the VMX file, I have also seen the VMX file become corrupt and be empty.

4. Export the VMDK file with VMKFSTOOL and then re-import it and "Create New VM" and add that disk as and existing disk.

5. If you have a backup of the VM, do a restore into a new VMDK file (Worst Case) I have seen VMDK's become corrupt, yet intermittently, due to issues with the LUN extents not being big enough and fragmenting the file too much... VMWare Support said they were corrupted at the LUN level and to do a restore and recreate the LUN... It sucked.

Hope this helps. Be sure to post what you find.

[heldstma]

Well, we seem to be back up. It turns out that simply migrating it to another ESX node fixed the problem (albeit probably not permanently).

[acmcnick]

If moving it to another host fixed it, make sure you time settings on all Hosts is the same. I have had permissions issues that seemed to be related to my hosts time drift was hours off. It caused failed vmotions as well. After I fixed the time everything went back to normal.

[Texiwill]

Hello,

If migrating it to another host fixed the problem then you could of had an existing lock on the VMDK that prevented it from being accessed. Given how the VM died/was killed, I would think this was the culprit. When you migrated it, it released the lock.

Best regards,

Edward

[heldstma]

Yesterday morning, it did the same thing, and I moved it BACK to the original ESX host. That fixed it again, too!

Hmmm...