Hello Togheter.. If got a bit a strange Prob. Enviroment: 2x ESX 5i on DL 380G2 Vcenter 5 on a VM W2k8R2 (up2date) On Each ESX is a HP P4300 VSA located with 2 Local VDMK's each arou...
See more...
Hello Togheter.. If got a bit a strange Prob. Enviroment: 2x ESX 5i on DL 380G2 Vcenter 5 on a VM W2k8R2 (up2date) On Each ESX is a HP P4300 VSA located with 2 Local VDMK's each around 1 TB The VSA Is located on a VSwitch2 The Switch has 2 Portgroups -iSan -ISCSI VMKernel On the Vswitch are on each ESX the Same Vnics (Numbre) connected. - vmnic 9 - vmnic 2 Both nics are connected to a stacked switch. The Portgroups are both configurated to use VLANID: 40 So the Switchports are Taget with the VLAND ID 40 an others. Ther is only a failover betwen the two Nics. So no LACP or things like this. The Kernel Port ist to comunicate with the ESX Hostsystem for Mounting Luns witch are provisioned on the VSA The ISAN is needet to connect to the other VSA Located on the Other ESX Server ESX1 ESX2 | | VM Kernel VM Kernel | | VSA1 <-Isan> VSA2 The HP VSA Is more or les also u2date (9.5.00.1215.0) Challange / Error The HP San Management has reported an error. The VSA2 was not Reachable during some Secconds. So i checked the Switch logs these where oke. So i checked the Logs form the VSA Feb 26 02:56:00 rep-4052-STO-002 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 26 02:56:15 rep-4052-STO-002 last message repeated 3 times So ther was a disconnect on the VSA. I checked the Vcenter logs. I found this: 2012-02-26T03:56:41.558+01:00 [07056 info 'Default' opID=HB-host-10@33510-b9989845] [VpxLRO] -- BEGIN task-internal-2050121 -- host-10 -- VpxdInvtHostSyncHostLRO.Synchronize -- 2012-02-26T03:56:41.558+01:00 [07056 info 'Default' opID=HB-host-10@33510-b9989845] [VpxdHostSync] Synchronizing host: host-10 (rep-4052-26-002.xxxx.local, 10.40.20.42) 2012-02-26T03:56:41.558+01:00 [02768 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 24851 ms 2012-02-26T03:56:41.558+01:00 [02788 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23306 ms 2012-02-26T03:56:41.558+01:00 [02472 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23306 ms 2012-02-26T03:56:41.558+01:00 [02444 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23306 ms 2012-02-26T03:56:41.558+01:00 [02944 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23306 ms 2012-02-26T03:56:41.558+01:00 [01608 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23306 ms 2012-02-26T03:56:41.558+01:00 [02796 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23306 ms 2012-02-26T03:56:41.558+01:00 [02568 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23291 ms 2012-02-26T03:56:41.558+01:00 [02792 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23072 ms 2012-02-26T03:56:41.558+01:00 [02800 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [02952 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [02560 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [02764 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [04876 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [02784 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [02552 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23057 ms 2012-02-26T03:56:41.558+01:00 [02736 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23041 ms 2012-02-26T03:56:41.558+01:00 [02556 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 23041 ms 2012-02-26T03:56:41.199+01:00 [02576 info 'Default' opID=SWI-3ce2c924] [ClientAdapterBase] InvokeOnSoap leaving 2012-02-26T03:56:41.558+01:00 [02576 info 'Default' opID=SWI-3ce2c924] [ClientAdapterBase] InvokeOnSoap leaving 2012-02-26T03:56:41.558+01:00 [04324 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 18080 ms hm oke so oke the Vcenter seams to have a prop to reach the ESX2 I traced the Log on the ESX in the vmkwarnig.log i found this: 2012-02-26T02:55:59.139Z cpu16:4112)WARNING: NetSched: 1713: Scheduler [0x41001d8facc0] lock up [stopped=0] for vmnic9: 2012-02-26T02:55:59.139Z cpu16:4112)WARNING: NetSched: 1723: detected at 7125337744 while last xmit at 7125332422 and 60/15114 packets/bytes in flight [window 2012-02-26T02:55:59.139Z cpu16:4112)WARNING: NetSched: 1732: Packets completion seem stuck, issuing reset on vmnic9 [stress 0] 2012-02-26T02:56:05.139Z cpu16:4112)WARNING: NetSched: 1713: Scheduler [0x41001d8facc0] lock up [stopped=0] for vmnic9: 2012-02-26T02:56:05.139Z cpu16:4112)WARNING: NetSched: 1723: detected at 7125343744 while last xmit at 7125337744 and 60/15114 packets/bytes in flight [window 2012-02-26T02:56:05.139Z cpu16:4112)WARNING: NetSched: 1732: Packets completion seem stuck, issuing reset on vmnic9 [stress 0] 2012-02-26T02:56:06.283Z cpu17:4515191)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000032" state in doubt; re 2012-02-26T02:56:06.283Z cpu17:4515191)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000034" state in doubt; re 2012-02-26T02:56:08.285Z cpu11:4515193)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000074" state in doubt; re 2012-02-26T02:56:10.286Z cpu17:4515195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000005b" state in doubt; re 2012-02-26T02:56:10.287Z cpu17:4515195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000028" state in doubt; re 2012-02-26T02:56:10.287Z cpu17:4515195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000062" state in doubt; re 2012-02-26T02:56:11.139Z cpu16:17103)WARNING: NetSched: 1713: Scheduler [0x41001d8facc0] lock up [stopped=0] for vmnic9: 2012-02-26T02:56:11.139Z cpu16:17103)WARNING: NetSched: 1723: detected at 7125349744 while last xmit at 7125343744 and 60/15114 packets/bytes in flight [windo 2012-02-26T02:56:11.139Z cpu16:17103)WARNING: NetSched: 1732: Packets completion seem stuck, issuing reset on vmnic9 [stress 0] 2012-02-26T02:56:11.669Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba36:CH:0 T:10 L:0 : Task mgmt "Abort Task" with itt=0x37369ab (refITT=0x37 2012-02-26T02:56:11.670Z cpu3:2509791)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000005b" state in doubt; req 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:58603 R: 10.40.40.52:3260] 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba36:CH:0 T:6 L:0 : Task mgmt "Abort Task" with itt=0x41b2240 (refITT=0x41b 2012-02-26T02:56:14.424Z cpu11:4515193)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; re 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:10 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:58607 R: 10.40.40.52:3260] 2012-02-26T02:56:14.424Z cpu4:4800)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reques 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:4 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:58601 R: 10.40.40.52:3260] 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:58601 R: 10.40.40.52:3260] 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba36:CH:0 T:4 L:0 : Task mgmt "Abort Task" with itt=0x2b83cc (refITT=0x2b8 2012-02-26T02:56:14.424Z cpu10:4515190)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000030" state in doubt; r 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:58598 R: 10.40.40.52:3260] 2012-02-26T02:56:14.424Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba36:CH:0 T:1 L:0 : Task mgmt "Abort Task" with itt=0x6cc925 (refITT=0x6cc 2012-02-26T02:56:14.424Z cpu17:4515191)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000036" state in doubt; r 2012-02-26T02:56:15.152Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; reque 2012-02-26T02:56:15.152Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reque 2012-02-26T02:56:15.179Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:8 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:15.179Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:15.179Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:58605 R: 10.40.40.52:3260] 2012-02-26T02:56:15.179Z cpu5:4800)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000065" state in doubt; reque 2012-02-26T02:56:15.179Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba36:CH:0 T:8 L:0 : Task mgmt "Abort Task" with itt=0x100d5be (refITT=0x10 2012-02-26T02:56:16.153Z cpu5:4101)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000065" state in doubt; reque 2012-02-26T02:56:16.188Z cpu4:4756)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; reque 2012-02-26T02:56:16.188Z cpu4:4756)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reque 2012-02-26T02:56:16.232Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:5 CN:0: Failed to receive data: Connection reset by peer 2012-02-26T02:56:16.232Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:16.232Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:52646 R: 10.40.40.52:3260] 2012-02-26T02:56:16.232Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:5 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6) 2012-02-26T02:56:16.232Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:16.232Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:52646 R: 10.40.40.52:3260] 2012-02-26T02:56:17.107Z cpu5:4515193)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000005b" state in doubt; re 2012-02-26T02:56:17.149Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; reque 2012-02-26T02:56:17.175Z cpu5:4101)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000065" state in doubt; reque 2012-02-26T02:56:17.176Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reque 2012-02-26T02:56:17.245Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:1 CN:0: Failed to receive data: Connection closed by peer 2012-02-26T02:56:17.245Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:17.245Z cpu23:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:61537 R: 10.40.40.50:3260] 2012-02-26T02:56:17.506Z cpu0:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:17.506Z cpu0:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:17.506Z cpu0:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:61537 R: 10.40.40.50:3260] 2012-02-26T02:56:17.771Z cpu0:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba36:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE" 2012-02-26T02:56:17.771Z cpu0:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:17.771Z cpu0:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.40.40.42:57769 R: 10.40.40.51:3260] 2012-02-26T02:56:18.032Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:8 CN:0: Failed to receive data: Connection closed by peer 2012-02-26T02:56:18.032Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:18.032Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:57936 R: 10.40.40.50:3260] 2012-02-26T02:56:18.144Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; reque 2012-02-26T02:56:18.159Z cpu5:4101)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000065" state in doubt; reque 2012-02-26T02:56:18.162Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reque 2012-02-26T02:56:18.293Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:8 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:18.293Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:18.293Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:57936 R: 10.40.40.50:3260] 2012-02-26T02:56:18.560Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba36:CH:0 T:8 CN:0: iSCSI connection is being marked "ONLINE" 2012-02-26T02:56:18.561Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052- 2012-02-26T02:56:18.561Z cpu14:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.40.40.42:52131 R: 10.40.40.52:3260] 2012-02-26T02:56:19.140Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; reque 2012-02-26T02:56:19.142Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reque 2012-02-26T02:56:19.600Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:5 CN:0: Failed to receive data: Connection closed by peer 2012-02-26T02:56:19.600Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:19.600Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:62245 R: 10.40.40.50:3260] 2012-02-26T02:56:19.858Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:5 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:19.858Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:19.858Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:62245 R: 10.40.40.50:3260] 2012-02-26T02:56:19.858Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:62245 R: 10.40.40.50:3260] 2012-02-26T02:56:20.119Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba36:CH:0 T:5 CN:0: iSCSI connection is being marked "ONLINE" 2012-02-26T02:56:20.119Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052- 2012-02-26T02:56:20.119Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.40.40.42:59185 R: 10.40.40.52:3260] 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:6 CN:0: Failed to receive data: Connection closed by peer 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:59779 R: 10.40.40.50:3260] 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:10 CN:0: Failed to receive data: Connection closed by peer 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:60874 R: 10.40.40.50:3260] 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba36:CH:0 T:4 CN:0: Failed to receive data: Connection closed by peer 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-405 2012-02-26T02:56:20.121Z cpu10:4798)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.40.40.42:55958 R: 10.40.40.50:3260] 2012-02-26T02:56:20.152Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc00000000000000048" state in doubt; reque 2012-02-26T02:56:20.170Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6000eb34fb8d7dc0000000000000006b" state in doubt; reque 2012-02-26T02:56:20.373Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:20.373Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:20.373Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:59779 R: 10.40.40.50:3260] 2012-02-26T02:56:20.374Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:10 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:20.374Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:20.374Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:60874 R: 10.40.40.50:3260] 2012-02-26T02:56:20.375Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba36:CH:0 T:4 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2012-02-26T02:56:20.375Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-st 2012-02-26T02:56:20.375Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.40.40.42:55958 R: 10.40.40.50:3260] 2012-02-26T02:56:20.642Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba36:CH:0 T:6 CN:0: iSCSI connection is being marked "ONLINE" 2012-02-26T02:56:20.642Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:20.642Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.40.40.42:56159 R: 10.40.40.52:3260] 2012-02-26T02:56:20.643Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba36:CH:0 T:10 CN:0: iSCSI connection is being marked "ONLINE" 2012-02-26T02:56:20.643Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:20.643Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.40.40.42:58186 R: 10.40.40.52:3260] 2012-02-26T02:56:20.643Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba36:CH:0 T:4 CN:0: iSCSI connection is being marked "ONLINE" 2012-02-26T02:56:20.643Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2003-10.com.lefthandnetworks:rep-4052-s 2012-02-26T02:56:20.643Z cpu6:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.40.40.42:49810 R: 10.40.40.52:3260] If i understand this correct there was a problem on the vmnic9 witch was autocorrected. But during this process the Connection betwen the two ESX (VSA) was broken...... Has any one a clu where the prob was... or wath i can do to pervent this problems in the future? Thanks for your time.. Kind Regards Florian