VMware Cloud Community
daniel_uk
Hot Shot
Hot Shot
‎01-03-2007 12:20 PM

ESX 3.01 Hang

Hi,

My ESX host totally froze today, it wasnt available on SSH or pingable. When sending an engineer to eyeball he said it had frozen.

This maybe a Server hardware issue but does anyone see anything in the logs that look suspicious?

Jan 2 15:10:39 ESXHOST vmkernel: 46:22:30:24.429 cpu0:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:10:39 ESXHOST last message repeated 3 times

Jan 2 15:10:39 ESXHOST vmkernel: 46:22:30:24.430 cpu0:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:10:39 ESXHOST last message repeated 7 times

Jan 2 15:10:39 ESXHOST vmkernel: 46:22:30:24.430 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:10:39 ESXHOST vmkernel: 46:22:30:24.431 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:10:39 ESXHOST last message repeated 2 times

Jan 2 15:11:41 ESXHOST vmkernel: 46:22:31:26.647 cpu0:1033)World: vm 11328: 690: Starting world vmware-vmx with flags 4

Jan 2 15:11:42 ESXHOST vmkernel: 46:22:31:27.495 cpu3:11328)World: vm 11329: 690: Starting world vmm0:VM with flags 8

Jan 2 15:11:42 ESXHOST vmkernel: 46:22:31:27.495 cpu3:11328)Sched: vm 11329: 4836: adding 'vmm0:VM': group 'host/user': cpu: shares=506 min=0 max=-1

Jan 2 15:11:42 ESXHOST vmkernel: 46:22:31:27.495 cpu3:11328)Sched: vm 11329: 4849: renamed group 2875 to vm.11328

Jan 2 15:11:42 ESXHOST vmkernel: 46:22:31:27.495 cpu3:11328)Sched: vm 11329: 4863: moved group 2875 to be under group 4

Jan 2 15:11:42 ESXHOST vmkernel: 46:22:31:27.509 cpu0:11328)Swap: vm 11329: 1426: extending swap to 512000 KB

Jan 2 15:11:43 ESXHOST vmkernel: 46:22:31:27.747 cpu0:11328)Migrate: vm 11329: 6838: Setting migration info ts = 4219571161, src ip = <172.16.1.242> dest ip = <0.0.0.0> Dest wid = -1

Jan 2 15:11:43 ESXHOST vmkernel: 46:22:31:27.747 cpu0:11328)World: vm 11330: 690: Starting world migSendHelper-11329 with flags 1

Jan 2 15:11:43 ESXHOST vmkernel: 46:22:31:27.747 cpu0:11328)World: vm 11331: 690: Starting world migRecvHelper-11329 with flags 1

Jan 2 15:11:43 ESXHOST vmkernel: 46:22:31:27.902 cpu3:1058)MigrateNet: vm 1058: 763: Accepted connection from <172.16.1.242>

Jan 2 15:11:54 ESXHOST vmkernel: 46:22:31:39.229 cpu2:11330)Alloc: vm 11329: 11547: Regular swap file bitmap checks out.

Jan 2 15:11:54 ESXHOST vmkernel: 46:22:31:39.266 cpu0:11331)Migrate: 3692: 4219571161: DONE paging in, status 0

Jan 2 15:11:54 ESXHOST vmkernel: 46:22:31:39.545 cpu0:11328)VSCSI: 2585: Creating Virtual Device for world 11329 vscsi0:0

Jan 2 15:11:54 ESXHOST vmkernel: 46:22:31:39.545 cpu0:11328)SCSI: 1271: Set shares value for world 11329 to 0x3e8

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.754 cpu1:11328)Migrate: 5914: 4219571161: No page at vpn 0x512 offset: 5316608 size: 16384 bytesLeft: 15095

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.806 cpu1:11328)World: vm 11332: 690: Starting world vmware-vmx with flags 44

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.808 cpu0:11332)World: vm 11333: 690: Starting world vmware-vmx with flags 44

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.809 cpu0:11332)World: vm 11334: 690: Starting world vmware-vmx with flags 44

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.809 cpu2:11329)Init: 677: Received INIT from world 11329

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.861 cpu3:11329)Migrate: 7056: 4219571161: Waiting to receive all changed pages before resuming...

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.861 cpu3:11329)Migrate: 7077: 4219571161: Received all changed pages.

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.861 cpu3:11329)Migrate: 3591: 4219571161: Resume handshake successful

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.861 cpu1:11330)World: vm 11330: 3864: Killing self with status=0x0:Success

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.861 cpu1:11331)World: vm 11331: 3864: Killing self with status=0x0:Success

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.861 cpu3:11329)World: vm 11335: 690: Starting world prefault-mig-11329 with flags 1

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.862 cpu2:11335)Swap: vm 11329: 5920: Starting opportunistic swapper for the migration swap file

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.898 cpu2:11335)Swap: vm 11329: 6120: Finish swapping in migration swap file. (faulted in 0 pages)

Jan 2 15:11:55 ESXHOST vmkernel: 46:22:31:39.898 cpu2:11335)World: vm 11335: 3864: Killing self with status=0x0:Success

Jan 2 15:15:39 ESXHOST vmkernel: 46:22:35:24.430 cpu0:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:15:39 ESXHOST vmkernel: 46:22:35:24.430 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:15:39 ESXHOST last message repeated 4 times

Jan 2 15:15:39 ESXHOST vmkernel: 46:22:35:24.431 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:15:39 ESXHOST last message repeated 9 times

Jan 2 15:20:39 ESXHOST vmkernel: 46:22:40:24.431 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:20:39 ESXHOST last message repeated 8 times

Jan 2 15:20:39 ESXHOST vmkernel: 46:22:40:24.432 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jan 2 15:20:39 ESXHOST last message repeated 6 times

0 Kudos
7 Replies
sbeaver
Leadership
Leadership
‎01-03-2007 12:25 PM

Tell me about the server and storage. What do you have and how is it configured?

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
whynotq
Commander
Commander
‎01-03-2007 12:31 PM

ran the error "bad000a I/O" through google and this thread pops up.

http://www.vmware.com/community/thread.jspa?messageID=537564

HP agents on the server seems to be the issue.

any help?

0 Kudos
daniel_uk
Hot Shot
Hot Shot
‎01-03-2007 01:35 PM

eek the logs are from yesterday...ive got logs from the right day;

I have from VMKWARNING;

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:00:33.546 cpu2:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:21. residual R 919, CR 0, ER 3

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:00:38.456 cpu2:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:00:38.456 cpu2:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:22. residual R 919, CR 0, ER 3

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:00:43.312 cpu2:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts[/i]

AND also from VMKWARNING, i think this was where I hardbooted it.

Jan 3 15:28:54 ESXHOST vmkernel: 0:00:05:49.777 cpu2:1075)WARNING: Cow: 1088: COW file was not closed cleanly, doing checks[/i]

VMKERNEL LOGS, these are cut down, i think it logged the same for every LUN i have.[/b]

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:01:12.922 cpu0:1024)Uplink: 2332: Disconnecting device vmnic2 from pps

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:01:12.922 cpu0:1024)Uplink: 2446: Connecting Port 0x2000003 to device vmnic2

Jan 3 08:19:26 ESXHOST vmkernel: 0:00:01:12.922 cpu0:1024)Uplink: 1291: Setting capabilities 0x0 for device vmnic2

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:12.922 cpu0:1024)Net: 1747: Opening discovery port

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:12.922 cpu0:1024)NetDiscover: 946: Using port 0x2000002's output chain

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:12.993 cpu0:1024)Uplink: 976: Couldn't find vSwitch2. Creating ps vSwitch2 of type 0xf

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:12.993 cpu0:1024)Net: 706: activating portset #3 as vSwitch2 with 64 ports, index mask is 0x3f

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.001 cpu0:1024)Uplink: 2332: Disconnecting device vmnic3 from pps

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.001 cpu0:1024)Uplink: 2446: Connecting Port 0x3000001 to device vmnic3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.001 cpu0:1024)Uplink: 1291: Setting capabilities 0x0 for device vmnic3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.001 cpu0:1024)Net: 1747: Opening discovery port

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.001 cpu0:1024)NetDiscover: 946: Using port 0x3000002's output chain

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.001 cpu0:1024)NetDiscover: 977: Inserting iocl

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.199 cpu0:1024)Loading module tcpip ...

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:13.201 cpu0:1024)Mod: 217: Starting load for module: tcpip R/O length: 0x3b000 R/W length: 0x1b000

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:14.312 cpu3:1027)<6>tg3: vmnic0: Link is up at 1000 Mbps, full duplex.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:14.312 cpu3:1027)<6>tg3: vmnic0: Flow control is on for TX and on for RX.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.152 cpu2:1033)Mod: 430: Module tcpip: initFunc: 0x82c618 text: 0x829000 data: 0x1f57d28 bss: 0x1f5b428
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.152 cpu2:1033)Mod: 446: mainHeap avail before: 13161008
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.152 cpu2:1033)Tcpip: 3640: portsetName = (null)
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip: 3225: Registered timer group = 1
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 2594: max mbufs: 8192 max clusters: 2048
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name socket size 288 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = socket max = 2048
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name udpcb size 224 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = udpcb max = 2048
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name inpcb size 224 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = inpcb max = 2048
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name tcpcb size 536 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = tcpcb max = 2048
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name tcptw size 48 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = tcptw max = 409
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip: 3579: support:tcp_timer_init
Jan 3 15:26:11 ESXHOST vmkernel:
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name syncache size 136 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = syncache max = 15359
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name sackhole size 16 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = sackhole max = 1024
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1839: name ripcb size 224 align 3 flags 0x20
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 1806: name = ripcb max = 2048
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 2607: Initializing routes
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 2615: rfc1323 \[High performance TCP extensions] is ENABLED
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Tcpip_Support: 2616: vmk_net_init complete
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Mod: 471: Initialization for tcpip succeeded.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.153 cpu2:1033)Module loaded successfully.
Jan 3 15:26:11 ESXHOST vmkernel:
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.158 cpu2:1033)Mod: 501: mainHeap avail after: 13152472
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.158 cpu2:1033)Mod: 509: no private ID set
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Uplink: 1291: Setting capabilities 0xb for device vmnic3
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Net: 1551: Setting Tx-complete cb for port
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Net: 1588: Setting cb for port
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Tcpip_Support: 2705: No NIC support for TSO
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Tcpip_Support: 2710: NIC support TX checksum offloading
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Tcpip_Support: 2716: NIC supports Scatter-Gather transmits
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Tcpip_Support: 2756: ether attach complete
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.243 cpu0:1024)Tcpip_Support: 2260: index = 8572400, ip_addr = 0xf30110ac, netmask = 0xfeffff
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.244 cpu0:1024)Tcpip_Support: 2306: error = 0
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.386 cpu0:1024)Loading module cosShadow ...
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.388 cpu0:1024)Mod: 217: Starting load for module: cosShadow R/O length: 0x3b000 R/W length: 0x1b000
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.425 cpu3:1027)<6>tg3: vmnic1: Link is up at 1000 Mbps, full duplex.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.425 cpu3:1027)<6>tg3: vmnic1: Flow control is on for TX and on for RX.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.537 cpu3:1027)<6>tg3: vmnic2: Link is up at 1000 Mbps, full duplex.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.537 cpu3:1027)<6>tg3: vmnic2: Flow control is on for TX and on for RX.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.551 cpu3:1027)<6>tg3: vmnic3: Link is up at 1000 Mbps, full duplex.
Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:15.551 cpu3:1027)<6>tg3: vmnic3: Flow control is on for TX and on for RX.

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.339 cpu2:1033)Mod: 430: Module cosShadow: initFunc: 0x867618 text: 0x864000 data: 0x1f74d40 bss: 0x1f78440

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.339 cpu2:1033)Mod: 446: mainHeap avail before: 13042168

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.339 cpu2:1033)Tcpip: 3640: portsetName = (null)

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip: 3225: Registered timer group = 2

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 2594: max mbufs: 4096 max clusters: 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name socket size 288 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = socket max = 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name udpcb size 224 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = udpcb max = 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name inpcb size 224 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = inpcb max = 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name tcpcb size 536 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = tcpcb max = 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name tcptw size 48 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = tcptw max = 204

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip: 3579: support:tcp_timer_init

Jan 3 15:26:11 ESXHOST vmkernel:

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name syncache size 136 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = syncache max = 15359

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name sackhole size 16 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = sackhole max = 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1839: name ripcb size 224 align 3 flags 0x20

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 1806: name = ripcb max = 1024

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 2607: Initializing routes

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 2615: rfc1323 \[High performance TCP extensions] is ENABLED

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Tcpip_Support: 2616: vmk_net_init complete

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Mod: 471: Initialization for cosShadow succeeded.

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.340 cpu2:1033)Module loaded successfully.

Jan 3 15:26:11 ESXHOST vmkernel:

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.345 cpu2:1033)Mod: 501: mainHeap avail after: 13033632

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.345 cpu2:1033)Mod: 509: no private ID set

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.609 cpu0:1024)VMNIX: VSI: 396: node 1076: get=d2106794 set=d2106844 list=d2106760 by ra=d2106613

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.609 cpu0:1024)VMNIX: VSI: 396: node 1077: get=d210684c set=d2106870 list=d180e580 by ra=d210662c

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.620 cpu0:1024)Uplink: 1291: Setting capabilities 0xb for device vmnic0

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.705 cpu0:1024)Loading module migration ...

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:17.708 cpu0:1024)Mod: 217: Starting load for module: migration R/O length: 0xe000 R/W length: 0x1000

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.194 cpu2:1033)Mod: 430: Module migration: initFunc: 0x8a8f00 text: 0x89f000 data: 0x1f91d68 bss: 0x1f91e48

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.194 cpu2:1033)Mod: 446: mainHeap avail before: 13029824

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.194 cpu2:1033)Migrate: 101: Registering migration module version 3.0

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.195 cpu2:1033)Migrate: 1379: Initializing migration module with module id 8...

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.195 cpu2:1033)Mod: 471: Initialization for migration succeeded.

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.195 cpu2:1033)Module loaded successfully.

Jan 3 15:26:11 ESXHOST vmkernel:

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.199 cpu2:1033)Mod: 501: mainHeap avail after: 13029104

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.199 cpu2:1033)Mod: 509: no private ID set

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.283 cpu0:1024)Loading module lvmdriver ...

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.285 cpu0:1024)Mod: 217: Starting load for module: lvmdriver R/O length: 0xc000 R/W length: 0x2000

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)Mod: 430: Module lvmdriver: initFunc: 0x8ad050 text: 0x8ad000 data: 0x1f92ef8 bss: 0x1f92f58

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)Mod: 446: mainHeap avail before: 13021216

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)FDS: 139: lvm

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1039: get=0x0 set=0x0 list=0x8b47e0 by ra=0x8b5061

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1046: get=0x0 set=0x0 list=0x8b484c by ra=0x8b50a8

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1056: get=0x0 set=0x0 list=0x8b4858 by ra=0x8b50e3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1038: get=0x8b48c4 set=0x8b48f4 list=0x0 by ra=0x8b5121

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1040: get=0x8b4a18 set=0x0 list=0x0 by ra=0x8b515f

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1041: get=0x8b4b64 set=0x8b4b94 list=0x0 by ra=0x8b519d

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1042: get=0x8b4bb8 set=0x8b4be8 list=0x0 by ra=0x8b51db

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1043: get=0x8b4c0c set=0x8b4c3c list=0x0 by ra=0x8b5219

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1044: get=0x8b4c60 set=0x8b4c90 list=0x0 by ra=0x8b5257

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1045: get=0x8b4d6c set=0x8b4d9c list=0x0 by ra=0x8b5295

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1047: get=0x8b4a18 set=0x0 list=0x0 by ra=0x8b52d3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1048: get=0x8b4b64 set=0x8b4b94 list=0x0 by ra=0x8b5311

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1049: get=0x8b4bb8 set=0x8b4be8 list=0x0 by ra=0x8b534f

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1050: get=0x8b4c0c set=0x8b4c3c list=0x0 by ra=0x8b538d

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1051: get=0x8b4c60 set=0x8b4c90 list=0x0 by ra=0x8b53cb

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1052: get=0x8b4d6c set=0x8b4d9c list=0x0 by ra=0x8b5409

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1054: get=0x8b4dcc set=0x8b4dfc list=0x0 by ra=0x8b5447

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1055: get=0x8b4ea4 set=0x8b4ed4 list=0x0 by ra=0x8b5485

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1057: get=0x8b4dc0 set=0x0 list=0x0 by ra=0x8b54c3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1058: get=0x8b4f8c set=0x8b4fbc list=0x0 by ra=0x8b5501

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:18.707 cpu2:1033)VSI: 396: node 1059: get=0x8b4fe0 set=0x8b5010 list=0x0 by ra=0x8b553f

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:19.616 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 64

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:20.600 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 48

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:21.585 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 32

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:22.561 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 16

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:23.567 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 0

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:23.567 cpu2:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:23.567 cpu2:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:21. residual R 919, CR 0, ER 3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:23.567 cpu2:1033)SCSI: 6608: Partition table read from device vmhba1:0:21 failed: SCSI reservation conflict (0xbad0022)

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:24.521 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 64

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:25.445 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 48

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:26.411 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 32

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:27.411 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 16

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:28.398 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 0

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:28.398 cpu2:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:28.398 cpu2:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:22. residual R 919, CR 0, ER 3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:28.398 cpu2:1033)SCSI: 6608: Partition table read from device vmhba1:0:22 failed: SCSI reservation conflict (0xbad0022)

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:29.332 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 64

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:30.245 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 48

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:31.253 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 32

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:32.289 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 16

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:33.291 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 0

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:33.291 cpu2:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:33.291 cpu2:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:23. residual R 919, CR 0, ER 3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:33.291 cpu2:1033)SCSI: 6608: Partition table read from device vmhba1:0:23 failed: SCSI reservation conflict (0xbad0022)

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:34.194 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 64

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:35.172 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 48

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:36.158 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 32

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:37.113 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 16

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:38.080 cpu2:1033)SCSI: vm 1033: 5509: Sync CR at 0

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:38.080 cpu2:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:38.080 cpu2:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:24. residual R 919, CR 0, ER 3

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:01:38.080 cpu2:1033)SCSI: 6608: Partition table read from device vmhba1:0:24 failed: SCSI reservation conflict (0xbad0022)

Jan 3 15:26:11 ESXHOST vmkernel: 0:00:0[/i]

Setup is HP BL25p with EVA 8000 on backend, this to me looks like its a SAN Connectivity issue but it dosnt explain why the COS was not accessible and the server not pingable.

Thanks for your help guys, im more likely to get a response than loggin a SR!

0 Kudos
masaki
Virtuoso
Virtuoso
‎01-06-2007 05:11 AM

As I told in that forum if you have HP Management Agents 7.6.0 try upgrading to 7.7.0.

0 Kudos
CKloseFUM
Enthusiast
Enthusiast
‎01-16-2007 07:06 AM

i can say this update from 7.6.0 - HP Agent 7.7.0 will help you

http://h18023.www1.hp.com/support/files/server/us/download/26407.html

i had those errors last week and my ESX-Server hang Smiley Sad

Only restart helped me for a while.

After updating the agent the error-messages that came in 5 minute frequency in the vmkwarning are gone Smiley Happy and everything works fine

0 Kudos
burnsl
Contributor
Contributor
‎03-20-2007 04:37 AM

oops posted, wrong forum..

Message was edited by:

burnsl

0 Kudos
multirotor
Contributor
Contributor
‎04-22-2007 11:49 PM

I still get this error on one of our ESX3.0.1 machines.

We are running hpmgmt 770 + ESX3.0.1 on 7 machines and only one of the machines gets this error.

It appears every 5 minutes:

Apr 23 08:35:47 vmw9 vmkernel: 4:16:10:12.250 cpu2:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Apr 23 08:40:47 vmw9 vmkernel: 4:16:15:12.249 cpu1:1040)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error[/i]

No problems so far, just a bit worried.

0 Kudos