strange iscsi / nic issue on esxi 6 / HP XW 8600

[GuenterKrembsle]

yes - i know that the HP xw8600 is not a supported system.

But i am using it as test system to learn how to troubleshoot an esxi infrastructure.

The strange thing is, that i have a functional software iscsi connection to a storage on a QNAP TS-809 U.

After some time (from some hours up to 14 days), the host is runnning into problems an it looks to me, as if there's a lost connection to the storage.

I tried to find the cause in the log files but allthough there's many error entries i am not really able to locate the source problem.

I am aware of the documents about "Troubleshooting iSCSI LUN connectivity issues ..." but it seems to me, as if the Network connection is the problem.

I found out, that i can not ping the destination in case of the error.

I tried to add a second nic as backup, but that does not solve the problem.

So my question is: How can i get information about the nic states via the ssh console?

What log files are available / relevant?

Thx for any help...

[GuenterKrembsle]

i was able to capture some mor info about the problem:

140371: 2016-03-27T13:48:03.411Z info hostd[65BB1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 172 : User root@192.168.0.5 logged out (login time: Sunday, 27 March, 2016 13:13:50, number of API invocations: 0, user agent: VMware-client)

140373: 2016-03-27T13:48:29.731Z error hostd[66740B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e960] CheckLicense: vFlash is not licensed. error = [N5Vmomi9DataArrayINS_18LocalizableMessageEEE:0x66a150a8]

140376: 2016-03-27T13:48:29.846Z error hostd[66740B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e961] CheckLicense: vFlash is not licensed. error = [N5Vmomi9DataArrayINS_18LocalizableMessageEEE:0x66670fc0]

140379: 2016-03-27T13:48:30.001Z info hostd[66740B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140410: 2016-03-27T13:48:30.002Z info hostd[66740B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 173 : Physical NIC vmnic3 linkstate is down.

140411: 2016-03-27T13:48:31.000Z info hostd[65BF2B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140450: 2016-03-27T13:48:31.002Z info hostd[65BF2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 174 : Lost network connectivity on virtual switch "vSwitch1". Physical NIC vmnic1 is down. Affected portgroups:"VMkernel".

140451: 2016-03-27T13:48:32.226Z error hostd[66740B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e969] CheckLicense: vFlash is not licensed. error = [N5Vmomi9DataArrayINS_18LocalizableMessageEEE:0x66175a40]

140454: 2016-03-27T13:48:32.267Z error hostd[65BB1B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e96a] CheckLicense: vFlash is not licensed. error = [N5Vmomi9DataArrayINS_18LocalizableMessageEEE:0x65c5f910]

140457: 2016-03-27T13:48:33.001Z info hostd[65BB1B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140488: 2016-03-27T13:48:33.002Z info hostd[65BB1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 175 : Physical NIC vmnic3 linkstate is up.

140489: 2016-03-27T13:48:33.699Z warning hostd[65BF2B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=HB-host-18@594-720cb766-b8-e974 user=vpxuser] GetVFlashResourceRuntimeInfo: vFlash is not licensed, not supported

140490: 2016-03-27T13:48:33.701Z info hostd[65BF2B70] [Originator@6876 sub=Hostsvc opID=HB-host-18@594-720cb766-b8-e974 user=vpxuser] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

140493: 2016-03-27T13:48:34.002Z info hostd[65BF2B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140532: 2016-03-27T13:48:34.003Z info hostd[65BF2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 176 : Network connectivity restored on virtual switch "vSwitch1", portgroups: "VMkernel". Physical NIC vmnic1 is up.

140533: 2016-03-27T13:48:39.837Z info hostd[65BB1B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140574: 2016-03-27T13:48:39.837Z info hostd[65BB1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 177 : Lost access to volume 56c89c4a-2b42a6c8-ea89-001a4b5386e1 (qnap-01) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

140575: 2016-03-27T13:48:50.454Z info hostd[66381B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140606: 2016-03-27T13:48:50.455Z info hostd[66381B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 178 : Device or filesystem with identifier 07d0560c-f61f4383 has entered the All Paths Down state.

140607: 2016-03-27T13:48:51.458Z info hostd[65ED2B70] [Originator@6876 sub=Hostsvc] VsanSystemVmkProvider : StorageHotPlugListener: VSAN service not configured to automatically claim storage: ignoring storage event

140608: 2016-03-27T13:48:51.458Z info hostd[66740B70] [Originator@6876 sub=Hostsvc] HandleVsanAutoClaimDiskEvent received auto-claim disk event, updating VSAN properties

140609: 2016-03-27T13:48:51.459Z info hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] VM Datastore updates for device file backing, refresh config

140610: 2016-03-27T13:48:51.459Z info hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] Send config update invoked

140611: 2016-03-27T13:48:51.501Z warning hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] UpdateStorageAccessibilityStatusInt: The datastore 192.168.0.229:ESXDataStore01 is not accessible

140612: 2016-03-27T13:48:51.501Z info hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to false

140613: 2016-03-27T13:48:51.503Z warning hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] FetchUpdatedLayout: VM storage inaccessible.

140614: 2016-03-27T13:48:51.504Z info hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] VM Datastore updates for device file backing, refresh config

140615: 2016-03-27T13:48:51.504Z info hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] Send config update invoked

140616: 2016-03-27T13:48:51.541Z warning hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] UpdateStorageAccessibilityStatusInt: The datastore 192.168.0.229:ESXDataStore01 is not accessible

140617: 2016-03-27T13:48:51.541Z info hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to false

140618: 2016-03-27T13:48:51.542Z warning hostd[65ED2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] FetchUpdatedLayout: VM storage inaccessible.

140619: 2016-03-27T13:48:52.994Z warning hostd[65BB1B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e995] GetVFlashResourceRuntimeInfo: vFlash is not licensed, not supported

140620: 2016-03-27T13:48:52.995Z info hostd[65BB1B70] [Originator@6876 sub=Hostsvc opID=8646e995] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

140623: 2016-03-27T13:48:56.969Z info hostd[667C2B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140654: 2016-03-27T13:48:56.973Z info hostd[667C2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 179 : Device or filesystem with identifier 07d0560c-f61f4383 has exited the All Paths Down state.

140655: 2016-03-27T13:48:57.272Z error hostd[66381B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e9a7] CheckLicense: vFlash is not licensed. error = [N5Vmomi9DataArrayINS_18LocalizableMessageEEE:0x668579b8]

140658: 2016-03-27T13:48:57.363Z error hostd[66381B70] [Originator@6876 sub=Hostsvc.VFlashManager opID=8646e9a8] CheckLicense: vFlash is not licensed. error = [N5Vmomi9DataArrayINS_18LocalizableMessageEEE:0x662180f8]

140661: 2016-03-27T13:48:57.982Z info hostd[649EFB70] [Originator@6876 sub=Hostsvc] VsanSystemVmkProvider : StorageHotPlugListener: VSAN service not configured to automatically claim storage: ignoring storage event

140662: 2016-03-27T13:48:57.983Z info hostd[65E91B70] [Originator@6876 sub=Hostsvc] HandleVsanAutoClaimDiskEvent received auto-claim disk event, updating VSAN properties

140663: 2016-03-27T13:48:57.983Z info hostd[649EFB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] VM Datastore updates for device file backing, refresh config

140664: 2016-03-27T13:48:57.983Z info hostd[649EFB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] Send config update invoked

140665: 2016-03-27T13:48:58.000Z info hostd[66381B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140696: 2016-03-27T13:48:58.001Z info hostd[66381B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 180 : Physical NIC vmnic3 linkstate is down.

140697: 2016-03-27T13:48:58.020Z info hostd[649EFB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-02/igk-02.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to true

140698: 2016-03-27T13:48:58.022Z info hostd[649EFB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] VM Datastore updates for device file backing, refresh config

140699: 2016-03-27T13:48:58.022Z info hostd[649EFB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] Send config update invoked

140700: 2016-03-27T13:48:58.061Z info hostd[649EFB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/56c89c4a-2b42a6c8-ea89-001a4b5386e1/igk-01/igk-01.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to true

140701: 2016-03-27T13:48:59.002Z info hostd[66740B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {

140740: 2016-03-27T13:48:59.003Z info hostd[66740B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 181 : Lost network connectivity on virtual switch "vSwitch1". Physical NIC vmnic1 is down. Affected portgroups:"VMkernel".

So i guess the nic causes the problem.

Well - i now tried to add a second nic to the switch to see, if this is changing something.

On the other hand i really wonder, why a connection loss to a datastore is able to block the esxi host.

I coulg see the console and "hostd" was not responding whil i saw on a different console window, that the system tried to connect to iscsi.