VMware Cloud Community
hammer1972
Enthusiast
Enthusiast
Jump to solution

VMware 7 not seeing LUNs

Hi All have a weird one here and hoping some has seen this before or has a bright idea.

Let me start at the begin to try and make sense of this issue and paint you a picture. I am running 3 x Lenovo SR635 servers stand alone, no VCenter to manage and all 3 are connected to a Lenovo DE4000H storage device. I recently upgraded our development server from VMware 6.7u3 to VMware 7.0U3 and all worked fine without any issues. I used the required VMware zip file from the Lenovo support site so that all necessary drives etc. are included.

I then continued to upgrade our first production server and all hell has broken lose. At first the upgrade appeared to be fine but I was unable to access the GUI VSphere interface from the browser using the IP or DNS name of the server. Logged ticket with Lenovo with little resolution, decided to re-install the server using a USB key. Again I used the ISO image from Lenovo support hoping for a speedy recovery. The network has now recovered and I can access GUI, but now I am unable to see any of the under lying LUNs from the DE4000H so that I can re-create my datastores. I have reconfigured the static targets and inputted the iqn... details, rescanned and refresh but nothing. Error logs do not show any error either.

I have followed a process found on the www about Network port bindings. I have removed this detail in the config and done a restart of the ESXi host but still I am unable to detect any LUN's. Does anyone have any idea why I am not able to see my LUN's using VMware 7.0U3?

PS the dev server that upgraded successfully sees the same LUN's that I am trying to connect to on the same VMware version. The only difference between the 2 servers is the BMC firmware and the UEFI version that Lenovo had suggested I upgrade to solve the original network issue. 

Any help or suggestions would be greatly appreciated.

Reply
0 Kudos
1 Solution

Accepted Solutions
hammer1972
Enthusiast
Enthusiast
Jump to solution

Hi

Finally found the issue, just thought I would put the update here for future.

Seems that during my clean up of all the network and storage components on the VMWare, caused me to get a new server initiator address, while adding the software ISCSI device back.

Once I had noticed the initiator name for the software ISCSI (vmhba64) did not correspond between the VMWare and the DE4000H device, I had to deleted the host from the DE4000H setup from the clustered hosts, re-added it back again with the new initiator address. Did a rescan on the VMware once more and can now we can see the LUNs that has allowed me to recreate the datastores.

Thanks all for your support and patients, you guys rock.

Now to get back to my original task,  upgrade to VMWare 7, fingers crossed it works smoothly. Or I'll be back 😉

View solution in original post

Reply
0 Kudos
14 Replies
e_espinel
Virtuoso
Virtuoso
Jump to solution

Hello.
From your description you are using an ISCSI connection.

Check the driver version that the vnic is using and the firmware version of the ethernet adapter.

# esxcli network nic list

# esxcli network nic get -n vmnicX

as an example

e_espinel_0-1673436270230.png

Compare with the development server that it is working fine.

On the Storage side (DE4000H) it is also necessary to validate if the current firmware supports version 6.7U3.

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
hammer1972
Enthusiast
Enthusiast
Jump to solution

Hi

I have compared the 2 servers and yes the drivers appear to be different. Not sure why this is the case, when I have used the customized VMware file from the Lenovo support page to upgrade both. As far as checking the DE4000H firmware. I assumed that the dev server was working, that firmware was supported. I could not find a affinitive answer if my older SAN firmware is officially supported or not, but seemed to work regardless.

Working Server:  vmnic0 and vmnic1                                                                   Broken Server vmnic0 and vmnic1

Driver Info:                                                                                                           Driver Info:
Bus Info: 0000:45:00:1                                                                                        Bus Info: 0000:41:00:1
Driver: i40en                                                                                                        Driver: i40en
Firmware Version: 6.01 0x8000363d 1.1824.0                                                    Firmware Version: 7.00 0x80005183 1.2203.0
Version: 2.2.4.0                                                                                                    Version: 2.3.4.0

DE4000H - Version 08.53.00.01.004-LEN  latest version on lenovo downloads seems to 08.74

If I upgrade the DE4000 firmware, will I have a similar issue on the dev server ? What is the correct order to update?

E.G. device driver, Server System Firmware, SAN firmware then VMWare ???

Problem I have I cannot afford for a 2nd server to not be available, as I have used the capacity of the other 2 remaining servers to keep the VMs up and running.

 

 

 

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Some thoughts:

  • check LUN presentation on the storage system, i.e. that the storage presents the LUNs to the new host iqn(s)?
  • make sure that MTU (in case you use Jumbo Frames) is set correctly on the iSCSI vSwitch(es), as well as on the port groups
  • verify connectivity between the iSCSI initiators and the targets using the vmkping utility, which allows you to specify the vmk, and the MTU size, e.g. vmkping -I vmk1 -d -s 8972 <target-IP>

If you cannot find anything, please provide details about the iSCSI setup (vSwitches, port groups, IP subnets, port binding, etc.)

André

Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso
Jump to solution

Hello.
On the working server the combination driver version 2.2.4.0 and Firmware 6.01 is working, but it would be difficult to get the Broken server to the same versions.

You can try to upgrade the Brocken server to the following combination Firmware 8.70 and driver 2.3.4.0

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
hammer1972
Enthusiast
Enthusiast
Jump to solution

Hi, thank you for your reply. I have check the below.

1. Check LUN presentation on the storage system, i.e. that the storage presents the LUNs to the new host iqn(s)?
Checked Storage device on both Controller A and Controller B. Both have the correct IP Address and the iqn number. Both show a link status of up.
2. make sure that MTU (in case you use Jumbo Frames) is set correctly on the iSCSI vSwitch(es), as well as on the port groups
Done, I have confirmed that both the vswitch and the adapters in question all have the same MTU size of 9000. As well as checking the MTU size on the ports of the storage device.
3. Verify connectivity between the iSCSI initiators and the targets using the vmkping utility, which allows you to specify the vmk,
and the MTU size, e.g. vmkping -I vmk1 -d -s 8972 <target-IP>
Both vmk1 and vmk2 pings return successfully.

vmkping -I vmk2 -d -s 8972 192.168.145.110
PING 192.168.145.110 (192.168.145.110): 8972 data bytes
8980 bytes from 192.168.145.110: icmp_seq=0 ttl=64 time=0.138 ms

vmkping -I vmk2 -d -s 8972 192.168.145.102
PING 192.168.145.102 (192.168.145.102): 8972 data bytes
8980 bytes from 192.168.145.102: icmp_seq=0 ttl=64 time=1.404 ms

4. If you cannot find anything, please provide details about the iSCSI setup (vSwitches, port groups, IP subnets, port binding, etc.)
vswitch
name: storage
MTU: 9000
uplink 1: vmnic0 - Up, 10000Mbps
uplink 2: vmnic1 - Up, 10000Mbps
Link Discovery
Mode: Listen
Protocol: Cisco discovery protocol (CDP)
Security
promiscuous: Reject
MAC address change: Accept
Forge Transmits: Accept
NIC teaming
Load Balancing: Route based on originating Port ID
Network failover detection: Link status only
Notify Switch: Yes
Failback: Yes
Failover Order: vmnic0, 10000Mbps,full duplex, Active
vmnic1, 10000Mbps,full duplex, Active
Traffic shaping: Disabled

Port Groups
Name          Virtual Switch Active Clients VLAN ID
StorageA      Storage                           1          0
StorageB       Storage                           1         0

VMK1
Port Group: StorageA
MTU: 9000
IPversion: IPv4 and IPv6          (Have tried to make it IPv4 only but never seems to keep setting)
IPv4 settings
configuration: Static
Address: 192.168.135.110
Subnet: 255.255.255.0
Services: vMotion, Provisioning, Fault tolerance logging, Replication, NFC replication

VMK2
Port Group: StorageB
MTU: 9000
IPversion: IPv4 and IPv6            (Have tried to make it IPv4 only but never seems to keep setting)
IPv4 settings
configuration: Static
Address: 192.168.145.110
Subnet: 255.255.255.0
Services: vMotion, Provisioning, Fault tolerance logging, Replication, NFC replication

Port Bindings (Have tried removing this from vmhba64 but makes no difference)
VMK0
Port Group: Management Network
MTU: 1500
IP Version: IPv4 and IPv6 (Have tried to make it IPv4 only but never seems to keep setting)
IPv4 settings
configuration: Static
Address: 172.16.xxx.xxx
Subnet: 255.255.255.0
Services: Management

#esxcli network nic get -n vmnic0
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 10000BaseCR4/Full
Auto Negotiation: false
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:41:00:0
Driver: i40en
Firmware Version: 7.00 0x80005183 1.2203.0
Version: 2.3.4.0
Link Detected: true
Link Status: Up
Name: vmnic0
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:52:13:a7
Wakeon: None

#esxcli network nic get -n vmnic1
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 10000BaseCR4/Full
Auto Negotiation: false
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:41:00:1
Driver: i40en
Firmware Version: 7.00 0x80005183 1.2203.0
Version: 2.3.4.0
Link Detected: true
Link Status: Up
Name: vmnic1
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:5f:77:45
Wakeon: None

Still not seeing any devices from the storage unit on vmware under Storage->Devices.

Any more thoughts of things to check.

PS: I tried removing everything to do with the storage, vswitch, port groups, vmhba etc. and add it all back again, was grasping at straws.

Now it will not save the 2nd static target under Storage->Adapters->vmhba64. No errors just not there after clicking save button and then checking again

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Ok, let's go through the iSCSI related settings:

With two VMKernel groups in different subnets,

VMK1
Port Group: StorageA
MTU: 9000
Address: 192.168.135.110

VMK2
Port Group: StorageB
MTU: 9000
Address: 192.168.145.110

each VMkernel port group has to be associated with a single, dedicated uplink (vmnic), i.e. one vmnic is "Active", and all other vmnics have to be set to "Unused" (not "Standby ! ).
Since the storage system is accessed using two different subnets, you must not configure Port Binding in the software iSCSI adapter, but only configure the dynamic/static targets.

As a side note: Check your storage vendor's documentation for how to configure "Delayed Ack". Many vendors recommend to disable "Delayed Ack" for better performance.

André

Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso
Jump to solution

Hello.
I once had problems configuring ISCSI, but changing my browser solved it (personally I use Waterfox G.X or Waterfox Clasic, which are based on Firefox).

Which model is your Lenovo SR635 7Y98 or 7Y99?

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
hammer1972
Enthusiast
Enthusiast
Jump to solution

Hi

Sorry for the delay, it has taken me a while to figure out how to do the upgrade. The 10GbE SFP adapters are now upgraded to a later version firmware.

esxcli network nic get -n vmnic0

Driver Info:
Bus Info: 0000:41:00:0
Driver: i40en
Firmware Version: 8.40 0x8000b67e 1.3079.0
Version: 1.16.3.0

esxcli network nic get -n vmnic1

Driver Info:
Bus Info: 0000:41:00:1
Driver: i40en
Firmware Version: 8.40 0x8000b67e 1.3079.0
Version: 1.16.3.0

However I still am unable to detect any LUN's on the DE4000H.

 

I should also mention we have re-installed version 6.7u3 from the Lenovo customised files. But get the same symptom. 

I have tried Firefox 108.0.2 (64-bit), Chrome Version 109.0.5414.74 and Edge Version 108.0.1462.76

 

Reply
0 Kudos
hammer1972
Enthusiast
Enthusiast
Jump to solution

I have split the vmkernel devices to each have there own Vswitch and port group. See settings below.

I have also rolled back to version 6.7 using the Lenovo customised ISO file and yet the problem persists.

VSwitch (StorageA)
MTU 9000
Uplink 1 vmnic0 - Up, 10000 mbps
Link discovery
Mode Listen
Protocol Cisco discovery protocol (CDP)
Security
Promiscuous mode Reject
MAC address changes Reject
Forged transmits Reject
NIC teaming
Load balancing Route based on originating port ID
Network failover detection Link status only
Notify switches Yes
Failback Yes
Failover order
Name Speed Status
vmnic0 10000 Mbps, full duplex Active
Traffic shaping
Status Disabled
Average bandwidth 100000 kb/s
Peak bandwidth 100000 kb/s
Burst size 102400 KB
Note
Traffic shaping policy is applied to the traffic of each virtual network adapter attached to the virtual switch.

VSwitch (StorageB)
MTU 9000
Uplink 1 vmnic1 - Up, 10000 mbps
Link discovery
Mode Listen
Protocol Cisco discovery protocol (CDP)
Security
Promiscuous mode Reject
MAC address changes Reject
Forged transmits Reject
NIC teaming
Load balancing Route based on originating port ID
Network failover detection Link status only
Notify switches Yes
Failback Yes
Failover order
Name Speed Status
vmnic1 10000 Mbps, full duplex Active
Traffic shaping
Status Disabled
Average bandwidth 100000 kb/s
Peak bandwidth 100000 kb/s
Burst size 102400 KB
Note
Traffic shaping policy is applied to the traffic of each virtual network adapter attached to the virtual switch.

vmk1
Port group StorageA
MTU 9000
IP version IPv4 and IPv6
IPv4 settings
Configuration Static
Address 192.168.133.110
Subnet mask 255.255.255.0
IPv6 settings Click to expand
TCP/IP stack Default TCP/IP stack
Services None ticked

vmk2
Port group StorageB
MTU 9000
IP version IPv4 and IPv6
IPv4 settings
Configuration Static
Address 192.168.143.110
Subnet mask 255.255.255.0
IPv6 settings Click to expand
TCP/IP stack Default TCP/IP stack
Services None ticked

Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso
Jump to solution

Hello.
If your network card is ThinkSystem Intel X710-T2L 10GBASE-T 2-Port PCIe Ethernet Adapter (4XC7A80266) then the driver version you should use is VMware ESXi 6.7 i40en 1.17.2.0 NIC Driver

For version 7.0, the driver must be  VMware ESXi 7.0 i40en 2.2.4.0 NIC Driver  or VMware ESXi7.0 i40en 2.3.4.0 NIC Driver.

attached link where to download it

https://customerconnect.vmware.com/downloads/details?downloadGroup=DT-ESXI67-INTEL-I40EN-11720&produ...

https://customerconnect.vmware.com/en/downloads/details?downloadGroup=DT-ESXI70-INTEL-I40EN-2240&pro...

https://customerconnect.vmware.com/en/downloads/details?downloadGroup=DT-ESXI70-INTEL-I40EN-2340&pro...

After making any changes you must run (at least 3 times) the rescan storage and HBA.
If your ethernet card is not as indicated, please send the full details of your vnic ethernet card.

If the problem continues, I offer you a free remote session to try to reconfigure the ISCSI on that ESXi host.
my time zone is UTC -5 to my personal email: eespinel@gmail.com, please contact me to coordinate date and time.

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
hammer1972
Enthusiast
Enthusiast
Jump to solution

Hi

I have gone to the URL you have supplied and down loaded the driver file (Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip) for VMware 6.7.

However when I use the command : esxcli software vib update -d /vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip  I get the below error:

[MetadataDownloadError]
Could not download from depot at zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml, skipping (('zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml', '', 'Error extracting index.xml from /vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip: "There is no item named \'index.xml\' in the archive"'))
url = zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml
Please refer to the log file for more details.

 

Checking the /var/log esxi.log file I see an entry

[2023-01-16 12:00:55,433 root ERROR] update failed: [MetadataDownloadError]
Could not download from depot at zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml, skipping (('zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml', '', 'Error extracting index.xml from /vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip: "There is no item named \'index.xml\' in the archive"'))
url = zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml
Please refer to the log file for more details.

I know the Lenovo installation likes you to have an XML file in the directory that is usually available with the file.

I have gone to Lenovo support to try and fine the same driver https://datacentersupport.lenovo.com/gb/en/products/servers/thinksystem/sr635/7y99/7y99cto1ww/j301zd... file VMware drivers package release for Intel Ethernet, but get the same result.

I have checked the readme file and it would appear I am doing the right thing, but yet it is not working.

I have verified the checksum and get the correct output.

 

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Although I don't think that it's a driver issue, note that the downloaded .zip archive contains the driver .zip that you need to extract.

ANdré

Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso
Jump to solution

Hello.
From the zip file you must extract the .vib file ( INT_bootbank_i40en_1.17.2.0-1OEM.670.0.0.8169922.vib)

then take it to the ESXi and run the command

esxcli software vib update -v {VIBFILE}

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
hammer1972
Enthusiast
Enthusiast
Jump to solution

Hi

Finally found the issue, just thought I would put the update here for future.

Seems that during my clean up of all the network and storage components on the VMWare, caused me to get a new server initiator address, while adding the software ISCSI device back.

Once I had noticed the initiator name for the software ISCSI (vmhba64) did not correspond between the VMWare and the DE4000H device, I had to deleted the host from the DE4000H setup from the clustered hosts, re-added it back again with the new initiator address. Did a rescan on the VMware once more and can now we can see the LUNs that has allowed me to recreate the datastores.

Thanks all for your support and patients, you guys rock.

Now to get back to my original task,  upgrade to VMWare 7, fingers crossed it works smoothly. Or I'll be back 😉

Reply
0 Kudos