ESX Hosts Losing Connection to iscsi SAN

[jakarth]

Hi all,

I have a problem which appears to be directly related to iscsi.

We have 2 ESX hosts connected to a HP MSA 1500i SAN via ISCSI with a gig switch inbetween.

Initially this worked fine (for about a week) but now I'm finding that every 20-30 minutes one of the ESX hosts seems to lose connectivity to the San. This seems to take out the other ESX host also. (I think this is because they are linked in VC under a datacentre grouping?)

I can only restore connectivity via rebooting the ESX servers. I have patched the troublesome server up to date with still no luck (the other node seems to be more stable) and I've even seen a kernel dump.

These are completely vanilla installs of ESX with nothing special configured at all except for iSCSI SAN connectivity.

To top this all off when we're building some Windows VM's and we try to log in (ctrl - alt - inst) they hang and lose all keyboard input. This then crashes the VI client aswell. I'm not sure if this is related or not

Can anyone offer any advice on ways to resolve this? I've not really used VMWare before (except for a little with FC) and I'm a bit out of my depth on this.

Any help would be greatly appreciated

Many Thanks

Jak

[Paul_Lalonde]

Fresh installs on HP hardware?

Make sure you're running the latest BIOS firmware on the servers, and the latest SAN firmware on the 1500i.

Then, post any relevant warnigns / errors in your /var/log/vmkernel file. Also post your /proc/vmware/pci and /proc/vmware/interrupts files.

Sounds like more of a hardware issue than anything else.

Paul

[oreeh]

and don't forget the harddisk firmware...

[jakarth]

Yeah they are fresh 3.0.1 installs on brand new HP kit.

I'll check out the error logs and post them when I'm back on site.

Thanks

Jak

[jakarth]

Hi everyone,

OK as requested I have checked the firmware for my San and Disks. The version is 1.32 which as far as I am aware is the latest HP MSA 1510i Firmware revision.

I have also extracted the logs as requested. I have attached them in his post.

There are two ESX hosts. ESX Host 1 is the server that seems to have hte most problems and is the one I have patched to try to rectify the issues (with no joy). ESX Host 2 is the server that appears to be working ok and is a vanilla install of ESX 3.0.1

ESX1[/b]

PCI file[/b]

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name

Spawned bus

000:00.0 8086:25d8 103c:31fd Host/PCI Intel C

000:02.0 8086:25e2 0000:0000 PCI/PCI Intel 009 C

000:03.0 8086:25e3 0000:0000 PCI/PCI Intel 006 C

000:04.0 8086:25f8 0000:0000 PCI/PCI Intel 019 C

000:05.0 8086:25e5 0000:0000 PCI/PCI Intel 022 C

000:06.0 8086:25e6 0000:0000 PCI/PCI Intel 002 C

000:07.0 8086:25e7 0000:0000 PCI/PCI Intel 004 C

000:16.0 8086:25f0 103c:31fd Host/PCI Intel C

000:16.1 8086:25f0 103c:31fd Host/PCI Intel C

000:16.2 8086:25f0 103c:31fd Host/PCI Intel C

000:17.0 8086:25f1 103c:31fd Host/PCI Intel C

000:19.0 8086:25f3 103c:31fd Host/PCI Intel C

000:21.0 8086:25f5 103c:31fd Host/PCI Intel C

000:22.0 8086:25f6 103c:31fd Host/PCI Intel C

000:29.0 8086:2688 103c:31fe USB Intel 5/ 16/0x79 A C

000:29.1 8086:2689 103c:31fe USB Intel 7/ 17/0x81 B C

000:29.2 8086:268a 103c:31fe USB Intel 10/ 18/0x89 C C

000:29.3 8086:268b 103c:31fe USB Intel 10/ 19/0x91 D C

000:29.7 8086:268c 103c:31fe USB Intel 5/ 16/0x79 A C

000:30.0 8086:244e 0000:0000 PCI/PCI Intel 001 C

000:31.0 8086:2670 0000:0000 PCI/ISA Intel C

000:31.1 8086:269e 103c:31fe IDE Intel 7/ / A C

001:03.0 1002:515e 103c:31fb Display ATI 7/ / A C

001:04.0 0e11:b203 103c:3305 0x880 Compaq 5/ 20/0xa1 A C

001:04.2 0e11:b204 103c:3305 0x880 Compaq 10/ 21/0xa9 B C

001:04.4 103c:3300 103c:3305 USB 0x103c 10/ 21/0xa9 B C

001:04.6 103c:3302 103c:3305 0xc07 0x103c 5/ 20/0xa1 A C

002:00.0 1166:0103 0000:0000 PCI/PCI SrvrWrks 003 C

003:00.0 14e4:164c 103c:7038 Ethernet Broadcom 10/ 18/0x89 A V bnx2 vmnic1

004:00.0 1166:0103 0000:0000 PCI/PCI SrvrWrks 005 C

005:00.0 14e4:164c 103c:7038 Ethernet Broadcom 10/ 19/0x91 A V bnx2 vmnic2

006:00.0 103c:3230 103c:3235 RAID 0x103c 5/ 16/0x79 A V cciss vmhba0

009:00.0 8086:3500 0000:0000 PCI/PCI Intel 010 C

009:00.3 8086:350c 0000:0000 PCI/PCI Intel 016 C

010:00.0 8086:3510 0000:0000 PCI/PCI Intel 011 C

010:01.0 8086:3514 0000:0000 PCI/PCI Intel 014 C

010:02.0 8086:3518 0000:0000 PCI/PCI Intel 015 C

011:00.0 8086:105e 103c:7044 Ethernet Intel 5/ 16/0x79 A V e1000 vmnic0

011:00.1 8086:105e 103c:7044 Ethernet Intel 7/ 17/0x81 B V e1000 vmnic3

Interrupts file[/b]

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3

0x21: 2 0 0 0 COS irq 1 (ISA edge),

0xa9: 64 0 0 0 COS irq 21 (PCI level)

0xdf: 55318888 55327911 55328262 55328262 VMK timer

0xe1: 0 0 0 0 VMK monitor

0xe9: 4198 8170 7898 44126 VMK resched

0xf1: 2 2 0 2 VMK tlb

0xf9: 205755 0 0 0 VMK noop

0xfc: 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 VMK error

0xff: 0 0 0 0 VMK spurious

VMKernel[/b]

Apr 3 18:05:34 vm-node1 vmkernel: TSC: 0 cpu0:0)Init: 260: cpu 0: early measured cpu speed 2666761476 Hz

Apr 3 18:05:34 vm-node1 vmkernel: TSC: 21088 cpu0:0)Cpu: 395: id1.version 6f6

Apr 3 18:05:34 vm-node1 vmkernel: TSC: 29512 cpu0:0)Cpu: 432: APIC ID mask: 0xff000000

Apr 3 18:05:34 vm-node1 vmkernel: TSC: 35520 cpu0:0)Cpu: 1127: initial APICID=0x0

Apr 3 18:05:35 vm-node1 vmkernel: TSC: 41480 cpu0:0)Cpu: 572: Update signature 4400000000, Platform ID 8000088240928

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 49904 cpu0:0)Cpu: 90: brand 0, type 2

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 53952 cpu0:0)Cpu: 91: numCores 2, numHT 1

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 58120 cpu0:0)Cpu: 92: cache size 0x400000, assoc. 16

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 63280 cpu0:0)Init: 269: vmkernelID not yet set.

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 172608 cpu0:0)VGA: 141: start b8000 end c0000

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 193520 cpu0:0)VGA: 152: 0x2c11000

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 198472 cpu0:0)VGA: 158: 8 screens

Apr 3 18:05:36 vm-node1 vmkernel: TSC: 203184 cpu0:0)VGA: 817: 16 pixels high

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 4350864 cpu0:0)Keyboard: 85: keyboard explicitly disabled

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 4365696 cpu0:0)VGA: 227: 1,2

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 7245072 cpu0:0)Term: 285: 0

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 7249328 cpu0:0)VGA: 221: 3

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 8687312 cpu0:0)Term: 285: 1

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 10530984 cpu0:0)VGA: 227: 4,5

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 13408656 cpu0:0)Term: 285: 2

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 19437784 cpu0:0)VGA: 221: 6

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 20876576 cpu0:0)Term: 285: 3

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 29926040 cpu0:0)Starting vmkernel initialization:

Apr 3 18:05:37 vm-node1 vmkernel: TSC: 30468664 cpu0:0)ACPI: 298: e1000: vmnic3: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.586 cpu2:1033)Mod: 430: Module vmfs3: initFunc: 0x8bb534 text: 0x89d000 data: 0x1f4db98 bss: 0x1f4df18

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.586 cpu2:1033)Mod: 446: mainHeap avail before: 13309376

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.586 cpu2:1033)World: vm 1053: 693: Starting world FS3ResMgr with flags 1

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.586 cpu2:1033)FSS: 265: Registered fs vmfs3, module 11, fsTypeNum 0xf530

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.686 cpu2:1033)Mod: 471: Initialization for vmfs3 succeeded.

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.686 cpu2:1033)Module loaded successfully.

Apr 3 18:06:03 vm-node1 vmkernel:

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.690 cpu2:1033)Mod: 501: mainHeap avail after: 13307288

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:38.690 cpu2:1033)Mod: 509: no private ID set

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:39.229 cpu2:1033)Config: 384: "ShowProgress" = 0, Old Value: 0, (Status: 0x0)

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.315 cpu0:1024)Tcpip_Support: 2326: dst = 0xfea9, netmask = 0xffff, gw = 0x0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.315 cpu0:1024)Tcpip: 2444: change route failed

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Net: 795: Enabling NIC in the shadow vmkernel tcpip stack

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip: 2289: Opening COSSharedPort port = 0x56dab88

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip_Support: 2705: No NIC support for TSO

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip_Support: 2712: No NIC support for checksum offloading

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip_Support: 2718: No NIC support for Scatter-Gather DMA

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip_Support: 2756: ether attach complete

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip_Support: 2260: index = 8629744, ip_addr = 0x0, netmask = 0x0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:40.481 cpu2:1033)Tcpip_Support: 2306: error = 0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.479 cpu0:1024)Tcpip_Support: 2260: index = 8629744, ip_addr = 0xb02010a, netmask = 0xff

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.479 cpu0:1024)Tcpip_Support: 2306: error = 0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.483 cpu0:1024)VMNIX: NetCos: 199: checking fea9,ffff,0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.483 cpu0:1024)Tcpip_Support: 2326: dst = 0x0, netmask = 0x0, gw = 0x0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.483 cpu0:1024)Tcpip: 2444: change route failed

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.486 cpu0:1024)VMNIX: NetCos: 199: checking fea9,ffff,0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.486 cpu0:1024)VMNIX: NetCos: 201: dup

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.486 cpu0:1024)Tcpip_Support: 2326: dst = 0xfea9, netmask = 0xffff, gw = 0x0

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.486 cpu0:1024)Tcpip: 2444: change route failed

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Net: 795: Enabling NIC in the shadow vmkernel tcpip stack

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip: 2289: Opening COSSharedPort port = 0x56e7ab8

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip_Support: 2705: No NIC support for TSO

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip_Support: 2712: No NIC support for checksum offloading

Apr 3 18:06:03 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip_Support: 2718: No NIC support for Scatter-Gather DMA

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip_Support: 2756: ether attach complete

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip_Support: 2260: index = 8629744, ip_addr = 0x0, netmask = 0x0

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:42.564 cpu2:1033)Tcpip_Support: 2306: error = 0

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:44.559 cpu0:1024)Tcpip_Support: 2260: index = 8629744, ip_addr = 0xb03010a, netmask = 0xff

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:44.559 cpu0:1024)Tcpip_Support: 2306: error = 0

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:44.564 cpu0:1024)VMNIX: NetCos: 199: checking fea9,ffff,0

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:44.564 cpu0:1024)VMNIX: NetCos: 201: dup

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:44.564 cpu0:1024)Tcpip_Support: 2326: dst = 0xfea9, netmask = 0xffff, gw = 0x0

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:45.790 cpu0:1024)Loading module vmfs2 ...

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:45.790 cpu0:1024)Mod: 217: Starting load for module: vmfs2 R/O length: 0x11000 R/W length: 0x11000

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.309 cpu2:1033)Mod: 430: Module vmfs2: initFunc: 0x8ce950 text: 0x8c0000 data: 0x1f4f0c8 bss: 0x1f4f2c8

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.309 cpu2:1033)Mod: 446: mainHeap avail before: 13238520

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.309 cpu2:1033)FSS: 265: Registered fs vmfs2, module 12, fsTypeNum 0xf520

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.562 cpu2:1033)Mod: 471: Initialization for vmfs2 succeeded.

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.562 cpu2:1033)Module loaded successfully.

Apr 3 18:06:04 vm-node1 vmkernel:

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.562 cpu2:1033)Mod: 501: mainHeap avail after: 13237208

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:46.562 cpu2:1033)Mod: 509: no private ID set

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:47.783 cpu0:1024)Loading module iscsi_mod ...

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:47.783 cpu0:1024)Mod: 217: Starting load for module: iscsi_mod R/O length: 0x2d000 R/W length: 0xa000

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)Mod: 430: Module iscsi_mod: initFunc: 0x8e73d0 text: 0x8d1000 data: 0x1f60130 bss: 0x1f60730

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)Mod: 446: mainHeap avail before: 13197072

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)linux_module_heap_init: Initial heap size : 65536, max heap size: 4194304

Apr 3 18:06:04 vm-node1 vmkernel:

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)iSCSI: 3.4.2 (16-Feb-2004) built for Linux 2.4.21-27.ELvmnix

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)iSCSI: will translate deferred sense to current sense on disk command responses

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)iSCSI: control device major number 254

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)LinPCI: 211: Device ffff:ffff claimed.

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)LinSCSI: 1282: vmhba40 is 65535:31.7 registered with key 0x1f6a340

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)iSCSI: allocating task cache iscsi_0000004822 with reaping disabled

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)iSCSI: detected HBA 0x1f6a340, host #0

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.096 cpu1:1034)LinSCSI: 389: NULL pci_dev

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.429 cpu1:1034)Mod: 471: Initialization for iscsi_mod succeeded.

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.429 cpu1:1034)Module loaded successfully.

Apr 3 18:06:04 vm-node1 vmkernel:

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.429 cpu1:1034)Mod: 501: mainHeap avail after: 13195088

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.429 cpu1:1034)Mod: 509: no private ID set

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.429 cpu0:1024)VMNIX: VmkDev: 1751: vmkiscsi

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:00:49.429 cpu0:1024)VMNIX: VmkDev: 1776: Registered 1 adapters

Apr 3 18:06:04 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)iSCSI: bus 0 target 0 = iqn.1986-03.com.hp.storage.msa1510i.sga0642034.target1

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)iSCSI: bus 0 target 0 portal 0 = address 10.1.3.23 port 3260 group 0

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)iSCSI: bus 0 target 0 portal 1 = address 10.1.3.24 port 3260 group 1

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)iSCSI: bus 0 target 0 portal 2 = address 10.1.3.25 port 3260 group 2

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)iSCSI: bus 0 target 0 portal 3 = address 10.1.3.26 port 3260 group 3

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)LinStubs: 1744: fn=0x8e1970 arg=0x0

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)World: vm 1054: 693: Starting world driver with flags 1

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu0:1054)LinStubs: 1813: fn=0x8e1970 arg=0x0

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)LinStubs: 1744: fn=0x8d9460 arg=0x3c583bf8

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu1:1034)World: vm 1055: 693: Starting world driver with flags 1

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.602 cpu0:1055)LinStubs: 1813: fn=0x8d9460 arg=0x3c583bf8

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.603 cpu1:1034)LinStubs: 1744: fn=0x8de908 arg=0x3c583bf8

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.603 cpu1:1034)World: vm 1056: 693: Starting world driver with flags 1

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.603 cpu2:1056)LinStubs: 1813: fn=0x8de908 arg=0x3c583bf8

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.603 cpu2:1056)iSCSI: bus 0 target 0 trying to establish session 0x3c583bf8 to portal 0, address 10.1.3.23 port 3260 group 0

Apr 3 18:06:05 vm-node1 vmkernel: 0:00:01:05.644 cpu2:1056)iSCSI: bus 0 target 0 established session 0x3c583bf8 #1 to portal 0, address 10.1.3.23 port 3260 group 0, alias target1

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 updating configuration of session 0x3c583bf8 to target1

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 = iqn.1986-03.com.hp.storage.msa1510i.sga0642034.target1

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 portal 0 = address 10.1.3.23 port 3260 group 0

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 portal 1 = address 10.1.3.24 port 3260 group 1

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 portal 2 = address 10.1.3.25 port 3260 group 2

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 portal 3 = address 10.1.3.26 port 3260 group 3

Apr 3 18:07:40 vm-node1 vmkernel: 0:00:02:54.235 cpu2:1033)iSCSI: bus 0 target 0 configuration updated at 17411, session 0x3c583bf8 to target1 does not need to logout

Apr 3 18:08:02 vm-node1 vmkernel: 0:00:03:16.128 cpu2:1033)SCSI: 8266: Starting rescan of adapter vmhba40

Apr 3 18:08:02 vm-node1 vmkernel: Vendor: HP Model: MSA1510i Rev: 1.32

Apr 3 18:08:02 vm-node1 vmkernel: Type: Unknown ANSI SCSI revision: 04

Apr 3 18:08:02 vm-node1 vmkernel: 0:00:03:16.128 cpu2:1033)LinSCSI: 4625: Device vmhba40:0:0 has appeared

Apr 3 18:08:02 vm-node1 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba40:0:0 : 0x0 0x80 0x83 0xc0

Apr 3 18:08:02 vm-node1 vmkernel: VMWARE SCSI Id: Device id info for vmhba40:0:0: 0x1 0x3 0x0 0x10 0x50 0x41 0x42 0x33 0x41 0x30 0x50 0x58 0x33 0x54 0x53 0x30 0x33 0x32 0x0 0x0 0x2 0x3 0x0 0x20 0x35 0x30 0x34 0x31 0x34 0x32 0x33 0x33 0x34 0x31 0x33 0x30 0x35 0x30 0x35 0x38 0x33 0x33 0x35 0x

Apr 3 18:08:02 vm-node1 vmkernel: 34 0x35 0x33 0x33 0x30 0x33 0x33 0x33 0x32 0x30 0x30 0x30 0x30

Apr 3 18:08:02 vm-node1 vmkernel: VMWARE SCSI Id: Id for vmhba40:0:0 0x50 0x41 0x42 0x33 0x41 0x30 0x50 0x58 0x33 0x54 0x53 0x30 0x33 0x32 0x00 0x00 0x4d 0x53 0x41 0x31 0x35 0x31

Apr 3 18:08:02 vm-node1 vmkernel: 0:00:03:16.130 cpu2:1033)SCSI: 1545: Device vmhba40:0:0 has not been identified as being attached to an active/passive SAN. It is either attached to an active/active SAN or is a local device.

Apr 3 18:08:02 vm-node1 vmkernel: Vendor: HP Model: MSA1510i VOLUME Rev: 1.32

Apr 3 18:08:02 vm-node1 vmkernel: Type: Direct-Access ANSI SCSI revision: 04

Apr 3 18:08:02 vm-node1 vmkernel: 0:00:03:16.130 cpu2:1033)LinSCSI: 4625: Device vmhba40:0:1 has appeared

Apr 3 18:08:02 vm-node1 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba40:0:1 : 0x0 0x80 0x83 0xc0 0xb0 0xc1

Apr 3 18:08:02 vm-node1 vmkernel: VMWARE SCSI Id: Device id info for vmhba40:0:1: 0x1 0x3 0x0 0x10 0x60 0x5 0x8 0xb3 0x0 0x93 0xa1 0xc0 0xcc 0x23 0x64 0x68 0x76 0xa4 0x0 0x2 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x20 0x36 0x30 0x30 0x35 0x30 0x38 0x42 0x33 0x30 0x30 0x39 0x33 0x41 0x3

Apr 3 18:08:02 vm-node1 vmkernel: 1 0x43 0x30 0x43 0x43 0x32 0x33 0x36 0x34 0x36 0x38 0x37 0x36 0x41 0x34 0x30 0x30 0x30 0x32 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x0

Apr 3 18:08:02 vm-node1 vmkernel: VMWARE SCSI Id: Id for vmhba40:0:1 0x60 0x05 0x08 0xb3 0x00 0x93 0xa1 0xc0 0xcc 0x23 0x64 0x68 0x76 0xa4 0x00 0x02 0x4d 0x53 0x41 0x31 0x35 0x31

Apr 3 18:08:02 vm-node1 vmkernel: 0:00:03:16.132 cpu2:1056)iSCSI: session 0x3c583bf8 recv_cmd 0x3c4047c8, cdb 0x1a, status 0x2, response 0x0, senselen 18, key 06, ASC/ASCQ 29/00, itt 12 task 0x3c580c68 to (0 0 0 1), target1

Apr 3 18:08:02 vm-node1 vmkernel: iSCSI: Sense 70000600 0000000a 00000000 29000000 0000

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.132 cpu2:1056)iSCSI: session 0x3c583bf8 recv_cmd 0x3c4047c8, itt 12, task 0x3c580c68 to (0 0 0 1), cdb 0x1a, underflow, received 0, residual 0, expected 256

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.132 cpu2:1033)SCSI: 8074: vmhba40:0:1:0 Retry (unit attn)

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.133 cpu2:1033)SCSI: 1545: Device vmhba40:0:1 has not been identified as being attached to an active/passive SAN. It is either attached to an active/active SAN or is a local device.

Apr 3 18:08:03 vm-node1 vmkernel: Vendor: HP Model: MSA1510i VOLUME Rev: 1.32

Apr 3 18:08:03 vm-node1 vmkernel: Type: Direct-Access ANSI SCSI revision: 04

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.134 cpu2:1033)LinSCSI: 4625: Device vmhba40:0:2 has appeared

Apr 3 18:08:03 vm-node1 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba40:0:2 : 0x0 0x80 0x83 0xc0 0xb0 0xc1

Apr 3 18:08:03 vm-node1 vmkernel: VMWARE SCSI Id: Device id info for vmhba40:0:2: 0x1 0x3 0x0 0x10 0x60 0x5 0x8 0xb3 0x0 0x93 0xa1 0xc0 0xad 0xd4 0x52 0x46 0xe0 0xac 0x0 0x4 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x20 0x36 0x30 0x30 0x35 0x30 0x38 0x42 0x33 0x30 0x30 0x39 0x33 0x41 0x3

Apr 3 18:08:03 vm-node1 vmkernel: 1 0x43 0x30 0x41 0x44 0x44 0x34 0x35 0x32 0x34 0x36 0x45 0x30 0x41 0x43 0x30 0x30 0x30 0x34 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x0

Apr 3 18:08:03 vm-node1 vmkernel: VMWARE SCSI Id: Id for vmhba40:0:2 0x60 0x05 0x08 0xb3 0x00 0x93 0xa1 0xc0 0xad 0xd4 0x52 0x46 0xe0 0xac 0x00 0x04 0x4d 0x53 0x41 0x31 0x35 0x31

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.136 cpu2:1056)iSCSI: session 0x3c583bf8 recv_cmd 0x3c4064e8, cdb 0x1a, status 0x2, response 0x0, senselen 18, key 06, ASC/ASCQ 29/00, itt 19 task 0x3c580c68 to (0 0 0 2), target1

Apr 3 18:08:03 vm-node1 vmkernel: iSCSI: Sense 70000600 0000000a 00000000 29000000 0000

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.136 cpu2:1056)iSCSI: session 0x3c583bf8 recv_cmd 0x3c4064e8, itt 19, task 0x3c580c68 to (0 0 0 2), cdb 0x1a, underflow, received 0, residual 0, expected 256

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.136 cpu2:1033)SCSI: 8074: vmhba40:0:2:0 Retry (unit attn)

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.137 cpu2:1033)SCSI: 1545: Device vmhba40:0:2 has not been identified as being attached to an active/passive SAN. It is either attached to an active/active SAN or is a local device.

Apr 3 18:08:03 vm-node1 vmkernel: Vendor: HP Model: MSA1510i VOLUME Rev: 1.32

Apr 3 18:08:03 vm-node1 vmkernel: Type: Direct-Access ANSI SCSI revision: 04

Apr 3 18:08:03 vm-node1 vmkernel: 0:00:03:16.138 cpu2:1033)LinSCSI: 4625: Device vmhba40:0:3 has appeared

Apr 3 18:08:03 vm-node1 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba40:0:3 : 0x0 0x80 0x83 0xc0 0xb0 0xc1

Apr 3 18:08:04 vm-node1 vmkernel: VMWARE SCSI Id: Device id info for vmhba40:0:3: 0x1 0x3 0x0 0x10 0x60 0x5 0x8 0xb3 0x0 0x93 0xa1 0xc0 0x7c 0xaa 0x7c 0xcb 0xde 0x7 0x0 0x5 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x20 0x36 0x30 0x30 0x35 0x30 0x38 0x42 0x33 0x30 0x30 0x39 0x33 0x41 0x31

Apr 3 18:08:04 vm-node1 vmkernel: 0x43 0x30 0x37 0x43 0x41 0x41 0x37 0x43 0x43 0x42 0x44 0x45 0x30 0x37 0x30 0x30 0x30 0x35 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x0

Apr 3 18:08:04 vm-node1 vmkernel: VMWARE SCSI Id: Id for vmhba40:0:3 0x60 0x05 0x08 0xb3 0x00 0x93 0xa1 0xc0 0x7c 0xaa 0x7c 0xcb 0xde 0x07 0x00 0x05 0x4d 0x53 0x41 0x31 0x35 0x31

Apr 3 18:08:04 vm-node1 vmkernel: 0:00:03:16.140 cpu2:1056)iSCSI: session 0x3c583bf8 recv_cmd 0x3c408328, cdb 0x1a, status 0x2, response 0x0, senselen 18, key 06, ASC/ASCQ 29/00, itt 26 task 0x3c580c68 to (0 0 0 3), target1

Apr 3 18:08:04 vm-node1 vmkernel: iSCSI: Sense 70000600 0000000a 00000000 29000000 0000

Apr 3 18:08:04 vm-node1 vmkernel: 0:00:03:16.140 cpu2:1056)iSCSI: session 0x3c583bf8 recv_cmd 0x3c408328, itt 26, task 0x3c580c68 to (0 0 0 3), cdb 0x1a, underflow, received 0, residual 0, expected 256

Apr 3 18:08:04 vm-node1 vmkernel: 0:00:03:16.140 cpu2:1033)SCSI: 8074: vmhba40:0:3:0 Retry (unit attn)

Apr 3 18:08:04 vm-node1 vmkernel: 0:00:03:16.141 cpu2:1033)SCSI: 1545: Device vmhba40:0:3 has not been identified as being attached to an active/passive SAN. It is either attached to an active/active SAN or is a local device.

Apr 3 18:08:05 vm-node1 vmkernel: 0:00:03:16.218 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 1 0)

Apr 3 18:08:05 vm-node1 vmkernel: 0:00:03:16.218 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 2 0)

Apr 3 18:08:05 vm-node1 vmkernel: 0:00:03:16.218 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 3 0)

Apr 3 18:08:05 vm-node1 vmkernel: 0:00:03:16.218 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 4 0)

Apr 3 18:08:05 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 5 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 6 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 7 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 8 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 9 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 10 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 11 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 12 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 13 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 14 0)

Apr 3 18:08:06 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 15 0)

Apr 3 18:08:07 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 16 0)

Apr 3 18:08:07 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 17 0)

Apr 3 18:08:07 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 18 0)

Apr 3 18:08:07 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 19 0)

Apr 3 18:08:07 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 20 0)

Apr 3 18:08:07 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 21 0)

Apr 3 18:08:08 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 22 0)

Apr 3 18:08:08 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 23 0)

Apr 3 18:08:08 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 24 0)

Apr 3 18:08:08 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 25 0)

Apr 3 18:08:08 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 26 0)

Apr 3 18:08:08 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 27 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 28 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 29 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 30 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.219 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 31 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 32 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 33 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 34 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 35 0)

Apr 3 18:08:09 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 36 0)

Apr 3 18:08:10 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 37 0)

Apr 3 18:08:10 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 38 0)

Apr 3 18:08:10 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 39 0)

Apr 3 18:08:10 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 40 0)

Apr 3 18:08:10 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 41 0)

Apr 3 18:08:10 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 42 0)

Apr 3 18:08:11 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 43 0)

Apr 3 18:08:11 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 44 0)

Apr 3 18:08:11 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 45 0)

Apr 3 18:08:12 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 46 0)

Apr 3 18:08:12 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 47 0)

Apr 3 18:08:12 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 48 0)

Apr 3 18:08:12 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 49 0)

Apr 3 18:08:13 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 50 0)

Apr 3 18:08:13 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 51 0)

Apr 3 18:08:13 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 52 0)

Apr 3 18:08:13 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 53 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 54 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 55 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 56 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 57 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 58 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.220 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 59 0)

Apr 3 18:08:14 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 60 0)

Apr 3 18:08:15 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 61 0)

Apr 3 18:08:15 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 62 0)

Apr 3 18:08:16 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 63 0)

Apr 3 18:08:16 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 64 0)

Apr 3 18:08:16 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 65 0)

Apr 3 18:08:16 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 66 0)

Apr 3 18:08:16 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 67 0)

Apr 3 18:08:16 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 68 0)

Apr 3 18:08:17 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 69 0)

Apr 3 18:08:17 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 70 0)

Apr 3 18:08:17 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 71 0)

Apr 3 18:08:17 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 72 0)

Apr 3 18:08:17 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 73 0)

Apr 3 18:08:17 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 74 0)

Apr 3 18:08:18 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 75 0)

Apr 3 18:08:18 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 76 0)

Apr 3 18:08:19 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 77 0)

Apr 3 18:08:19 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 78 0)

Apr 3 18:08:19 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 79 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 80 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 81 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 82 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 83 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 84 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 85 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.221 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 86 0)

Apr 3 18:08:20 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 87 0)

Apr 3 18:08:21 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 88 0)

Apr 3 18:08:21 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 89 0)

Apr 3 18:08:21 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 90 0)

Apr 3 18:08:21 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 91 0)

Apr 3 18:08:22 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 92 0)

Apr 3 18:08:22 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 93 0)

Apr 3 18:08:22 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 94 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 95 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 96 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 97 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 98 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 99 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 100 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 101 0)

Apr 3 18:08:23 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 102 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 103 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 104 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 105 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 106 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 107 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 108 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 109 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.222 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 110 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 111 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 112 0)

Apr 3 18:08:24 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 113 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 114 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 115 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 116 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 117 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 118 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 119 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 120 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 121 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 122 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 123 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 124 0)

Apr 3 18:08:25 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 125 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 126 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 127 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 128 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 129 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 130 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 131 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 132 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 133 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 134 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 135 0)

Apr 3 18:08:26 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 136 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 137 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.223 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 138 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 139 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 140 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 141 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 142 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 143 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 144 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 145 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 146 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 147 0)

Apr 3 18:08:27 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 148 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 149 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 150 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 151 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 152 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 153 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 154 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 155 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 156 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 157 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 158 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 159 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 160 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 161 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 162 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 163 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 164 0)

Apr 3 18:08:28 vm-node1 vmkernel: 0:00:03:16.224 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 165 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 166 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 167 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 168 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 169 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 170 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 171 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 172 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 173 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 174 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 175 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 176 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 177 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 178 0)

Apr 3 18:08:29 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 179 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 180 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 181 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 182 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 183 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 184 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 185 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 186 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 187 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 188 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 189 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 190 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 191 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.225 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 192 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 193 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 194 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 195 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 196 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 197 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 198 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 199 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 200 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 201 0)

Apr 3 18:08:30 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 202 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 203 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 204 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 205 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 206 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 207 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 208 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 209 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 210 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 211 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 212 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 213 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 214 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 215 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 216 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 217 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 218 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.226 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 219 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 220 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 221 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 222 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 223 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 224 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 225 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 226 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 227 0)

Apr 3 18:08:31 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 228 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 229 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 230 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 231 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 232 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 233 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 234 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 235 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 236 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 237 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 238 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 239 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 240 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 241 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 242 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.227 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 243 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 244 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 245 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 246 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 247 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 248 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 249 0)

Apr 3 18:08:32 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 250 0)

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 251 0)

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 252 0)

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 253 0)

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 254 0)

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)iSCSI: queuecommand 0x3c408a48 failed to find a session for HBA 0x1f6a340, (0 0 255 0)

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)SCSI: 8323: Finished rescan of adapter vmhba40

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:16.228 cpu0:1033)SCSI: 8203: Disks have been added or removed from the system.

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:17.599 cpu0:1024)IOAPIC: 1080: 0x59 retriggerred

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:22.994 cpu2:1033)World: vm 1057: 693: Starting world vmware-vmkauthd with flags 4

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:41.771 cpu3:1033)World: vm 1058: 693: Starting world migServer with flags 1

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:41.772 cpu3:1033)Migrate: 1573: Successfully enabled migration support.

Apr 3 18:08:33 vm-node1 vmkernel: 0:00:03:41.772 cpu3:1033)Config: 384: "Enabled" = 1, Old Value: 0, (Status: 0x0)

ESX2[/b]

PCI File[/b]

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name

Spawned bus

000:00.0 8086:25d8 103c:31fd Host/PCI Intel C

000:02.0 8086:25e2 0000:0000 PCI/PCI Intel 009 C

000:03.0 8086:25e3 0000:0000 PCI/PCI Intel 006 C

000:04.0 8086:25f8 0000:0000 PCI/PCI Intel 019 C

000:05.0 8086:25e5 0000:0000 PCI/PCI Intel 022 C

000:06.0 8086:25e6 0000:0000 PCI/PCI Intel 002 C

000:07.0 8086:25e7 0000:0000 PCI/PCI Intel 004 C

000:16.0 8086:25f0 103c:31fd Host/PCI Intel C

000:16.1 8086:25f0 103c:31fd Host/PCI Intel C

000:16.2 8086:25f0 103c:31fd Host/PCI Intel C

000:17.0 8086:25f1 103c:31fd Host/PCI Intel C

000:19.0 8086:25f3 103c:31fd Host/PCI Intel C

000:21.0 8086:25f5 103c:31fd Host/PCI Intel C

000:22.0 8086:25f6 103c:31fd Host/PCI Intel C

000:29.0 8086:2688 103c:31fe USB Intel 3/ 16/0x69 A C

000:29.1 8086:2689 103c:31fe USB Intel 4/ 17/0x71 B C

000:29.2 8086:268a 103c:31fe USB Intel 5/ 18/0x79 C C

000:29.3 8086:268b 103c:31fe USB Intel 7/ 19/0x81 D C

000:29.7 8086:268c 103c:31fe USB Intel 3/ 16/0x69 A C

000:30.0 8086:244e 0000:0000 PCI/PCI Intel 001 C

000:31.0 8086:2670 0000:0000 PCI/ISA Intel C

000:31.1 8086:269e 103c:31fe IDE Intel 4/ / A C

001:03.0 1002:515e 103c:31fb Display ATI 10/ / A C

001:04.0 0e11:b203 103c:3305 0x880 Compaq 10/ 20/0x91 A C

001:04.2 0e11:b204 103c:3305 0x880 Compaq 7/ 21/0x99 B C

001:04.4 103c:3300 103c:3305 USB 0x103c 7/ 21/0x99 B C

001:04.6 103c:3302 103c:3305 0xc07 0x103c 10/ 20/0x91 A C

002:00.0 1166:0103 0000:0000 PCI/PCI SrvrWrks 003 C

003:00.0 14e4:164c 103c:7038 Ethernet Broadcom 5/ 18/0x79 A V bnx2 vmnic1

004:00.0 1166:0103 0000:0000 PCI/PCI SrvrWrks 005 C

005:00.0 14e4:164c 103c:7038 Ethernet Broadcom 7/ 19/0x81 A V bnx2 vmnic2

006:00.0 103c:3230 103c:3235 RAID 0x103c 3/ 16/0x69 A V cciss vmhba0

009:00.0 8086:3500 0000:0000 PCI/PCI Intel 010 C

009:00.3 8086:350c 0000:0000 PCI/PCI Intel 016 C

010:00.0 8086:3510 0000:0000 PCI/PCI Intel 011 C

010:01.0 8086:3514 0000:0000 PCI/PCI Intel 014 C

010:02.0 8086:3518 0000:0000 PCI/PCI Intel 015 C

011:00.0 8086:105e 103c:7044 Ethernet Intel 3/ 16/0x69 A V e1000 vmnic0

011:00.1 8086:105e 103c:7044 Ethernet Intel 4/ 17/0x71 B V e1000 vmnic3

Interrupts File[/b]

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3

0x21: 2 0 0 0 COS irq 1 (ISA edge),

0x99: 64 0 0 0 COS irq 21 (PCI level)

0xdf: 54797958 54807048 54805118 54805118 VMK timer

0xe1: 193 270 362 374 VMK monitor

0xe9: 275337 472724 410265 335022 VMK resched

0xf1: 6330 9516 7018 9601 VMK tlb

0xf9: 229549 0 0 0 VMK noop

0xfc: 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 VMK error

0xff: 0 0 0 0 VMK spurious

Vmkernel File[/b]

Is an empty file with no content.

I hope this gives you guys some idea of what's going on

I have run the esxcfg-firewall -e swISCSIClient command as I have read that configuring iSCSI via the VI client doesn't sort out the firewall but apart from that I've not done anything else.

Thanks in advance

Jak

[Paul_Lalonde]

You've got IRQ sharing happening between your USB subsystem and your GigE and SmartArray devices. Disable USB completely in the BIOS and bring your ESX server backup. Allow it to reconfigure hardware and reboot. I'm seeing error messages regarding interrupt steering in your log file as well as interrupt vectors shared between the service console and vmkernel.

Paul

[jakarth]

Thanks Paul I will give this a try

[jakarth]

Well I disabled USB which was fine.

The problem still occurs however now it's slightly different.

The ESX hosts now find the iSCSI SAN everytime which is great, the problem is that they constantly lose the connection and then pick it back up again. This can be once every few hours or once every few minutes.

VMWare support have not told me anything yet other than they see the network going up and down.

All my kit is new, I have a dedicated network and switch for iSCSI and I've no idea why this is occuring as there's no packet loss when pinging hosts on this switch etc

[jakarth]

The plot thickens...

I've made numerous configuration changes to my SAN and network but there's still no joy with this.

To cut a long story short what is happening is that the 2 ESX hosts pick up the iscsi san fine on inital boot. They work happily away until more than 1 host tries to write to the SAN at the same time (well it appears this way at least) sometimes everything is ok for days other times it falls over straight away. The symptoms are that the hosts lose connectivity to the SAN, everything crashes upon reboot normal service is restored.

There's nothing wrong hardware wise here as I've checked everything down to the bare cables.

As part of my testing I removed 1 host from using iSCSi. I left it in the datacentre config within Virtual Centre but disabled iscsi via the vi client. As soon as I powered on a vm local to this servers disk however it caused my other esx host (connected to the iscsi san) to lose connectivity to it's disk.

I'm fastly running out of ideas now and any help would be greatly appreciated, even some pointers as to how to get a temporary work around would be something.

Thanks in advance

J

[jakarth]

OK well,

I think I have resolved the problem.

VMWare ESX Version 3.0.1 does not support dual controller configurations for iSCSI. This includes Active / Passive configurations.

On top of this they do not support dual storage processor configurations, the only failover you can have with iSCSI configurations is on the nics within Virtual Centre.

VMWare advised me of both of these in my support call yet I couldn't find any reference in any product documentation.....

So we disabled our standby controller, and disabled the 2nd storage port on our controller unit. Thus leaving 1 controller with 1 active port allocated 1 ip address.

This same solitary IP was entered into the dynamic discovery tab of the storage adapters of the 2 ESX hosts. After rebooting both they picked up iSCSI with no problems and appear to be working.

If they stay up for longer than a week then I'll assume all is ok and this has resolved it.

it's worth mentioning also that VMWare advised me to move the iSCSI connectivity away from the on board broadcom nics to the intel based additional cards I had installed which I did.

Fingers crossed this will resolve it

[oreeh]

it's worth mentioning also that VMWare advised me to

move the iSCSI connectivity away from the on board

broadcom nics to the intel based additional cards I

had installed which I did.

did they say why ?

[BUGCHK]

yet I couldn't find any reference in any product documentation.....

http://h18000.www1.hp.com/products/quickspecs/12230_div/12230_div.html

What's New[/b]

\* Support for VMware (single controller only)

...

OS Support[/b]

\* VMWare ESX 3.0.1 (single controller only)

[jakarth]

What wasn't clear to me was that fact that it didn't support

Active / Passive configurations (ie one live controller and one standby)

I realised it wouldn't support Active / Active configurations but I couldn't believe we had to physically disable the passive controller. But now I know.

Also the fact we had to disable one of the ports on the controller was certainly news to me as I couldn't find that in any document.

Thanks for the links though I think now I know more than most about MSA 1510i iSCSI Sans and VMWare

To answer the question regarding the nics, VMWare told me that potentially there could be issues with the broadcom driver so told me to switch it around, they didn't go into any further detail though I'm afraid.

[jakarth]

OK well I've had no problems since I've made the changes listed in this post and from what vmware advised.

Thanks to all for your help.

[admin]

Yeah we're looking at the MSA1510i and I noticed the HCL state single controller only, bummer. It's not exactly clear but the little 1 is next to the 1510i.

http://www.vmware.com/pdf/vi3_san_guide.pdf

How have you configured the paths to the storage? I couldn't work out how to even get network redundancy, you assign an IP to each NIC on the single controller, but you can only specificy one IP path int he s/w iSCSI initiator, if you add the second IP the host see's the LUNs twice - and assumes the second instances of the LUNs are snapshot volumes so disables access (good job too or you'd no doubt screw the volumes).

Could you let me have your thoughts on how it performs as well? Are you using an MSA20 or MSA30 disk shelf?

[jakarth]

Well after consultation with vmware we disabled the 2nd nic on the controller as we faced the same problem. There appears to be no way to configure redundancy with iscsi except with the network cards on the actual esx hosts. The san itself is single point of failure city (I'm happy to be proved otherwise!)

As for performance, well it's not too bad actually now that we don't have the san connectivity dropping all the time! We're using an MSA20 shelf populated with 500GB Sata 7200rpm disks and it's perfectly fine for our test lab.

We're replicating an exchange migration so we have 3 domains a a couple of exchange servers and it's fine thus far though we haven't really hammered it to be honest. I guess it depends what you want to use it for but for a test rig it's perfectly acceptable.

[Empel]

We seem to experience the same or very similar issues with our setup. I am particularly curious about how disabling the second NIC on the Ethernet controller helped in your case. In our setup we use the first NIC only for management (MA0), the dataport is left unconfigured. The second NIC is used only for data (SA1), this NIC is connected to the dedicated iSCSI switch. This setup in itself works fine, we have no problem doing discovery or sending data, but once every few days the VMhosts lose connectivity and only physically disconnecting and reconnecting the Ethernet cables solves the issue apart from more drastic measures like completely rebooting the hosts. One of our hosts uses Intel cards to access the iSCSI switch, the other uses Broadcom cards. They seem to go down together all the time, like in your case.

JW

[jakarth]

Sadly not being a vmware guru I'm not 100% sure why this fixed the issue, however what appeared to be happening was that having 2 nics allocated to the iscsi network seemed to cause conflicts. It appeared that some requests would be sent from one card and then their responses received on the other, by removing one of the nics we guaranteed that sent and received packets always went to and from the same network card.

So we had a scenario where our controller had 1 active network card connected to the VMware iSCSi network and the management port was allocated an ip on this same network which we accessed via a web browser on a dedicated SAN management vm on the same network.