iSCSI SAN 101

[Kahonu84]

Aloha kakou,

I am trying to get my ESX 3.01 server to see our Lefthand iSCSI SAN. In the end, I am going nowhere fast. The bottom line is I'm not sure where to begin with troubleshooting. I went thru all the steps outline in the LH docs. However I do not see any LUNs.

From the ESX console, we tried to ping the SAN - plugging the cable into each of the 8 network ports and could not see it. The cable is known to be good.

Mahalo in advance,

Bill

[Dave_Mishchenko]

Are you using the ESX software initiator and if so how is the networking setup on the virtual switch that connects to the SAN.

[Kahonu84]

Are you using the ESX software initiator and if so

how is the networking setup on the virtual switch

that connects to the SAN.

Yes, software. I have VSwitch3 set up with VMKernal and Service Console ports. VSwitch3 is then connected to VMNIC3.

[Paul_Lalonde]

Please post the output from:

esxcfg-nics -l

esxcfg-vmknic -l

esxcfg-vswitch -l

esxcfg-vswif -l

What is the IP address of the vSwitch3 service console? the VMkernel interface? the LH iSCSI SAN appliance?

Thanks

Paul

[jeffpurcell]

A very useful command line utility to validate VMkernel and IP storage network layer access is vmkping.

It provides standard ping like functionality for the VMkernel IP interface to the iSCSI target or portal group.

Once you have established the network connectivity is ok, you can

allow the iSCSI software initiator from the firewall settings,

enable the iSCSI client service from the ESX server

define the iSCSI target discovery address

This should allow you to establish and iSCSI session with the storage device, you will still need to associate and mask the storage devices to the IQN of the ESX software initiator in order to see the LUNs which you've defined for the ESX host.

[Kahonu84]

NICS -

Name PCI Driver Link Speed Duplex Description

vmnic0 05:00.00 bnx2 Up 100Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic1 09:00.00 bnx2 Up 100Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic2 0a:00.00 e1000 Up 100Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic3 0a:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic4 0c:00.00 e1000 Up 100Mbps Half Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic5 0c:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic6 0e:00.00 e1000 Up 100Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic7 0e:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

VMKNIC

Port Group IP Address Netmask Broadcast MAC Address MTU Enabled

VSWITCH

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch0 32 3 32 vmnic0

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

VM Network portgroup1 0 0 vmnic0

Service Console portgroup0 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch1 64 2 64 vmnic1

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC1-DMZ portgroup3 0 0 vmnic1

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch2 64 3 64 vmnic2

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC2-Internalportgroup4 0 1 vmnic2

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch4 64 2 64 vmnic4

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC4-DMZ portgroup6 0 0 vmnic4

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch5 64 4 64 vmnic5

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC5-Internalportgroup7 0 2 vmnic5

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch6 64 3 64 vmnic6

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC6-Internalportgroup8 0 1 vmnic6

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch7 64 3 64 vmnic7

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC7-Internalportgroup9 0 1 vmnic7

VSWIF

Name Port Group IP Address Netmask Broadcast Enabled DHCP

vswif0 Service Console xxx.yyy.5.6 255.255.252.0 xxx.yyy.7.255 true false

IP of vswitch3 service console - 10.1.1.84

IP of vswitch3 VMkernal interface - 10.1.1.85

IP of SAN - xxx.yyy.98.31

[Paul_Lalonde]

I don't see a VMkernel interface anywhere, and mysteriously you're missing vSwitch3...

You need a VMkernel interface for iSCSI to work properly.

Paul

[Kahonu84]

Let me try again shortly. We deleted it in order to try something else. I will put it back the way it was and repost.

[Kahonu84]

NICS

Name PCI Driver Link Speed Duplex Description

vmnic0 05:00.00 bnx2 Up 100Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic1 09:00.00 bnx2 Up 100Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic2 0a:00.00 e1000 Up 100Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic3 0a:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic4 0c:00.00 e1000 Up 100Mbps Half Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic5 0c:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic6 0e:00.00 e1000 Up 100Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic7 0e:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

VMKNIC

Port Group IP Address Netmask Broadcast MAC Address MTU Enabled

VMkernel 10.1.1.85 255.0.0.0 10.255.255.255 00:50:56:69:1e:3d 1514 true

VSWIF

Name Port Group IP Address Netmask Broadcast Enabled DHCP

vswif0 Service Console xxx.yyy.5.6 255.255.252.0 xxx.yyy.7.255 true false

vswif1 Service Console 2 10.1.1.84 255.0.0.0 10.255.255.255 true false

VSWITCH

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch0 32 3 32 vmnic0

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

VM Network portgroup1 0 0 vmnic0

Service Console portgroup0 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch1 64 2 64 vmnic1

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC1-DMZ portgroup3 0 0 vmnic1

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch2 64 3 64 vmnic2

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC2-Internalportgroup4 0 1 vmnic2

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch4 64 2 64 vmnic4

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC4-DMZ portgroup6 0 0 vmnic4

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch5 64 4 64 vmnic5

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC5-Internalportgroup7 0 2 vmnic5

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch6 64 3 64 vmnic6

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC6-Internalportgroup8 0 1 vmnic6

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch7 64 3 64 vmnic7

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC7-Internalportgroup9 0 1 vmnic7

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch3 64 4 64 vmnic3

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

VMkernel portgroup21 0 1 vmnic3

Service Console 2 portgroup20 0 1 vmnic3

[Anders_Gregerse]

Everytime you change something concerning iSCSI configuration on ESX you might need to reboot the host (just my experience). We are switching to HBAs to avoid those problems.

I'm a very big ESX fan, but the iSCSI software initiator seems not quite finished. I had to disable and enable it through commandline, it completely ignored what I did through the GUI. And compared to how easy it is with FC, iSCSI should be as easy.

[Paul_Lalonde · Accepted solution]

Your VMkernel, service console, and IP SAN appliance \*should* all be on the same subnet. They don't appear to be that way.

Paul

[conreyta]

I've got iSCSI working and I stumbled on a few things along the way. We have a service console port in network 1, that is used for everything but iSCSI.

The "storage network" is an isolated network that has it's own Service Console Port, as well as the VMkernel port for iSCSI.

It seemed odd at first that iSCSI requires two IP addresses in order to get working for a single host, however, it's working well now. Not getting a Service console port in the same "network/subnet" for the iSCSI VMkernel port caused a few headaches for us getting started.

-Theron

[Kahonu84]

Ok. I took care of that and rebooted. Still can't see the SAN. Config now is -

NICS -

Name PCI Driver Link Speed Duplex Description

vmnic0 05:00.00 bnx2 Up 100Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic1 09:00.00 bnx2 Up 100Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic2 0a:00.00 e1000 Up 100Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic3 0a:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic4 0c:00.00 e1000 Up 100Mbps Half Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic5 0c:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic6 0e:00.00 e1000 Up 100Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic7 0e:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

VMKNIC -

Port Group IP Address Netmask Broadcast MAC Address MTU Enabled

VMkernel xxx.yyy.98.132 255.255.255.0 xxx.yyy.98.255 00:50:56:69:1e:3d 1514 true

VSWIF -

Name Port Group IP Address Netmask Broadcast Enabled DHCP

vswif0 Service Console xxx.yyy.5.6 255.255.252.0 xxx.yyy.7.255 true false

vswif1 Service Console 2 xxx.yyy.98.131 255.255.255.0 xxx.yyy.98.255 true false

VSWITCH -

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch0 32 3 32 vmnic0

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

VM Network portgroup1 0 0 vmnic0

Service Console portgroup0 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch1 64 2 64 vmnic1

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC1-DMZ portgroup3 0 0 vmnic1

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch2 64 3 64 vmnic2

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC2-Internalportgroup4 0 1 vmnic2

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch4 64 2 64 vmnic4

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC4-DMZ portgroup6 0 0 vmnic4

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch5 64 4 64 vmnic5

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC5-Internalportgroup7 0 2 vmnic5

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch6 64 3 64 vmnic6

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC6-Internalportgroup8 0 1 vmnic6

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch7 64 3 64 vmnic7

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

ESX1-VMNIC7-Internalportgroup9 0 1 vmnic7

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch3 64 4 64 vmnic3

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

Service Console 2 portgroup20 0 1 vmnic3

VMkernel portgroup21 0 1 vmnic3

[Paul_Lalonde]

Can you vmkping the IP of the SAN array?

Paul

[Kahonu84]

Yes sir. Couldn't do that before.

[Paul_Lalonde]

\- Have you enabled the software iSCSI service to communicate through the firewall? (Configuration tab, Security Profile, Properties, check the Software iSCSI Client option)

If your LH SAN authorizes the client host by IQN name:

\- Have you configured your LH SAN to permit access to the IQN name of the ESX server's iSCSI initiator?

If your LH SAN authorizes the client host by IP address:

\- Have you added both the service console IP and the VMkernel IP to the list of allowed iSCSI initiators on the LH SAN?

Paul

[Kahonu84]

Yes, the firewall was configured.

Yes, the SAN admin gave me access via the ICQ name ESX spit out.

Bill

[Kahonu84]

Mr. Paul - I get 'em!!!! After redoing the vswitch, I wiped out everything done after that point previously and re-did everything. And boom - I got a pretty lil' SAN volume. Let's hope it's fast enuf to be usable.

Mahalo nui for all your help.

Bill