ESXi 6.0 U1 to U2 upgrade, iSCSI issues

[Gthr33]

Hello,

First time posting, so I'll try and sum up my thoughts and what I've done with troubleshooting.  Please let me know if I left anything out or more information is needed.

I'm running the free ESXi 6.0 on a Supermicro X10SLL-F mATX board with an Intel Xeon E3-1231v3 CPU and 32 GB of DDR3 ECC UDIMMs.  I use a 4G USB flash drive for boot, 75 GB 2.5" SATA for local storage (i.e. /scratch), and a portion of a 120 GB SSD for host cache as well as local storage.  The main datastore for the VMs are located on on an iSCSI target (running FreeNAS 9.3.x).  This setup has worked very well since the install of ESXi 6.0 (June 2015), then 6.0 U1 (Sept. 2015) and I just recently took the plunge to 6.0 U2.  I figured everything should be business as usual for the upgrade...

However, after upgrading to 6.0 U2 none of the iSCSI volumes are "seen" by ESXi anymore -- vmhba38:C0:T0 & vmhba38:C0:T1, although I can confirm that I can ping the iSCSI target and that the NIC (vmnic1), vSwitch, and VMware iSCSI Software Adapter are loaded -- I have not made any changes to either ESXi nor the iSCSI host prior to the upgrade to 6.0 U2.  It was all working beforehand.

I next went digging into logs; vmkernel.log and vobd.log all report that it couldn't contact the storage due to a network issue.  I also did some standard network troubleshooting (see VMware KB 1008083); everything passed, except for the jumbo frame test.

[root@vmware6:~] tail –f /var/log/vmkernel.log | grep iscsi

[ ... ]

2016-03-31T05:05:48.217Z cpu0:33248)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba38:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:5)

2016-03-31T05:05:48.217Z cpu0:33248)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID:  TARGET: (null) TPGT: 0 TSIH: 0]

2016-03-31T05:05:48.217Z cpu0:33248)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.xxx.yyy.195:41620 R: 10.xxx.yyy.109:3260]

2016-03-31T05:05:48.218Z cpu4:33248)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba38:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:5)

2016-03-31T05:05:48.218Z cpu4:33248)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID:  TARGET: (null) TPGT: 0 TSIH: 0]

2016-03-31T05:05:48.218Z cpu4:33248)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.xxx.yyy.195:32715 R: 10.xxx.yyy.109:3260]

[root@vmware6:~] tail –f /var/log/vobd.log

[ ... ]

2016-03-31T05:05:48.217Z: [iscsiCorrelator] 1622023006us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba38:C0:T0

2016-03-31T05:05:48.217Z: [iscsiCorrelator] 1622023183us: [vob.iscsi.target.connect.error] vmhba38 @ vmk1 failed to login to  iqn.2005-10.org.freenas.ctl:vmware-iscsi because of a network connection failure.

2016-03-31T05:05:48.217Z: [iscsiCorrelator] 1622002451us: [esx.problem.storage.iscsi.target.connect.error] Login to iSCSI target iqn.2005-10.org.freenas.ctl:vmware-iscsi on vmhba38 @ vmk1 failed. The iSCSI initiator could not establish a network connection to the target.

2016-03-31T05:05:48.218Z: [iscsiCorrelator] 1622023640us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba38:C0:T1

2016-03-31T05:05:48.218Z: [iscsiCorrelator] 1622023703us: [vob.iscsi.target.connect.error] vmhba38 @ vmk1 failed to login to

[root@vmware6:~] ping 10.xxx.yyy.109

PING 10.xxx.yyy.109 (10.xxx.yyy.109): 56 data bytes

64 bytes from 10.xxx.yyy.109: icmp_seq=0 ttl=64 time=0.174 ms

64 bytes from 10.xxx.yyy.109: icmp_seq=1 ttl=64 time=0.238 ms

64 bytes from 10.xxx.yyy.109: icmp_seq=2 ttl=64 time=0.309 ms

--- 10.xxx.yyy.109 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.174/0.240/0.309 ms

[root@vmware6:~] vmkping 10.xxx.yyy.109

PING 10.xxx.yyy.109 (10.xxx.yyy.109): 56 data bytes

64 bytes from 10.xxx.yyy.109: icmp_seq=0 ttl=64 time=0.179 ms

64 bytes from 10.xxx.yyy.109: icmp_seq=1 ttl=64 time=0.337 ms

64 bytes from 10.xxx.yyy.109: icmp_seq=2 ttl=64 time=0.382 ms

--- 10.xxx.yyy.109 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.179/0.299/0.382 ms

[root@vmware6:~] nc -z 10.xxx.yyy.109 3260

Connection to 10.xxx.yyy.109 3260 port [tcp/*] succeeded!

[root@vmware6:~] vmkping -s 8972 10.xxx.yyy.109 -d

PING 10.xxx.yyy.109 (10.xxx.yyy.109): 8972 data bytes

--- 10.xxx.yyy.109 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

I then started looking at the NIC drivers thinking maybe something got screwed up during the upgrade; not the first time I've seen issues with the out-of-box drivers provided by VMware.  I checked VMware's HCL for I/O devices; the physical NICs used on this host are Intel I217-LM (nic-e1000e), Intel I210 (nic-igb), and Intel 82574L (nic-e1000e).  VMware's HCL lists that the driver for the I217-LM & 82574L should be version 2.5.4-6vmw and I210 should be 5.0.5.1.1-5vmw.  When I went to check, I noticed it was using a different version for the e1000e driver (I210 driver version was correct).

[root@vmware6:~] esxcli software vib list | grep e1000e

Name                           Version                               Vendor  Acceptance Level  Install Date

-----------------------------  ------------------------------------  ------  ----------------  ------------

net-e1000e                     3.2.2.1-1vmw.600.1.26.3380124         VMware  VMwareCertified   2016-03-31

The esxupgrade.log would seem to indicate that VMware's e1000e 2.5.4-6vmw should have been loaded...

[root@esxi6-lab:~] grep e1000e /var/log/esxupdate.log

[ ... ]

### ESXi 6.0 U1 upgrade

2015-09-29T22:20:29Z esxupdate: BootBankInstaller.pyc: DEBUG: About to write payload 'net-e100' of VIB VMware_bootbank_net-e1000e_2.5.4-6vmw.600.0.0.2494585 to '/tmp/stagebootbank'

[ … ]

### ESXi 6.0 U2 upgrade

2016-03-31T03:47:24Z esxupdate: BootBankInstaller.pyc: DEBUG: About to write payload 'net-e100' of VIB VMware_bootbank_net-e1000e_2.5.4-6vmw.600.0.0.2494585 to ''/tmp/stagebootbank'

The e1000e 3.2.2.1-1vmw is the recommended driver for ESXi 5.5 U3 and not ESXi 6.0 U2!  Since these drivers are listed as "Inbox", I'm not sure if there an easy path to download vendor-provided drivers (vib) for it, or if they exist even.  I found an article on-line to manually update drivers on an ESXi host using esxcli; I attempted to check for and install any newer e1000e drivers.

[root@vmware6:~] esxcli software vib update -n net-e1000e -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

Installation Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed: VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.78.3248547

   VIBs Removed: VMware_bootbank_net-e1000e_3.2.2.1-1vmw.600.1.26.3380124

   VIBs Skipped:

As you can see it found a newer version and installed it.  However after rebooting it still did not fix the issue.  I even went so far as to force the CHAP password used to authenticate by resetting it and updating it on both sides (iSCSI initiator & target).  At this point I'm wondering if I need to somehow downgrade and use the 2.5.4-6vmw driver (how?) or if there is some other issue at play here.  Am I going down a rabbit hole with my idea that it's a NIC driver issue?

Thanks in advance.

-- G

[Gthr33 · Accepted solution]

I found [a|the?] solution: downgrade to the Intel e1000e 2.5.4-6vmw drivers from 6.0 U1.  See VMware KB 2124715 for the file name.

Steps:

1. Log into https://my.vmware.com/group/vmware/patch#search
2. Select "ESXi (Embedded and Installable)" and "6.0.0".  Select the Search button.
3. Scroll down and look for the Release Name update-from-esxi6.0-6.0_update01 (released 9/10/2015) . Put a check mark next to it and then Download button to the right.
   
4. Save the file somewhere locally. Open the ZIP archive after it downloads.
5. Navigate to the "vib20" directory. Extract the net-e1000e folder.  Contained within is the Intel e1000e drivers from ESXi 6.0 U1: VMware_bootbank_net-e1000e_2.5.4-6vmw.600.0.0.2494585.vib
   
6. Transfer this to your ESXi server (either via SCP or even the file browser within the vSphere Thick Client). Save it somewhere you'll remember (i.e. /tmp).
7. Login via SSH to the ESXi 6.0 host.
8. Issue the command to install it:  esxcli software vib install -v /tmp/VMware_bootbank_net-e1000e_2.5.4-6vmw.600.0.0.2494585.vib
   
9. Reboot the ESXi host.  Once it comes back online your iSCSI datastore(s)/volumes should return.  You can verify by issuing df -h or esxcli storage core device list at the CLI prompt.  vSphere Client also works
   

It took some time to find the right file(s), steps and the commands to use.  I hope someone else can benefit from this.  I like what VMware can provide for virtualization, but lately it seems their QA department has been out to lunch.

Still do not have a definitive explanation as to why the Intel e1000e drivers from ESXi 5.5 U3 were used when upgrading from ESXi 6.0 U1 to U2.  Perhaps someone with more insight or VMware Support person can.

&out

-- G

[oetti]

You made my day!

Thanks

Chris

[Yarisken]

You also made my day .... damn the time i spent to solve this 🙂

@

[virtualDD]

Thank you, sir! We faced iSCSI issues with two different customers, both looked the same, but different storage vendors and versions both times (one patched from 5.5 u2 to u3 and one from 5.5 u2 to 6.0u2). Following this fixed the issue where both strorage vendor and vmware support failed.