ESXi Host can't format LUNs on iSCSI

[tzimmermann]

Dear Community,

this problem is driving crazy...

We have a cluster of 4 ESXi 5.1 Hosts managed with vCenter 5.1, all latest patch level.

Connected to the Hosts are via Software iSCSI our Active/Passive iSCSI cluster, which is working just fine, and two small iSCSI NAS Systems for minor stuff like Install Images, some backups and so on.

Now, all hosts are configured the same, using profiles, all can without problem access the SAN and the first NAS, but not the second.

These are the steps I took for the first AND the second NAS:

• create RAID5
• create LUNs
• set a iSCSI user and password
• check if the network connection is valid (it is, for both NAS!)
• configure the "Send target" on the ESXi Hosts
• Check that the targets are showing up (they are, 3 for each NAS, making 6 of them!)
• Then add the storage, chosing the filesystem and so on.

And here is where NAS 1 shows up and I can add all the LUNs, format them to VMFS and use them on all hosts, whereas NAS 2 LUNs don't show up.

The NAS systems are configured exactly the same, same hardware, same firmware, they only differ by IP, username and password.

In the NAS 2 system when I click on the LUN it says that the ESXi-Hosts are connected and the LUN connection is active, just like in NAS 1.

So, where could be the problem?

Any hints are highly apreciated 😉

Best regards,

Thomas

[rickardnobel]

On ESXi, on Storage Adapters, if you select the vmhba for iSCSI, do you see all LUNs? Or only from the first SAN/NAS?

Have you set the permissions of the second NAS for which IPs (or IQNs) that has read/write access to the LUNs?

[tzimmermann]

On Storage Adapters I only see the "Devices" of NAS 1 and our SAN, not NAS 2.

NAS 1 and 2 are simple NAS without masking capabilities, you can just select username and password or no authentication for iSCSI.

I tried mountin one of the 3 LUNs from a Windows host in order to check if it is working, and it is, without problem. Once the connection is established the NAS 2 shows an active connection for the LUN. And when I configure the ESXi hosts, it does so, too.

(of course I severed the connection from the Windows host after the test!)

[tzimmermann]

This is what NAS 2 is stating for the  LUNs:

initiator:iqn.1998-01.com.vmware:esxi4-50e6a774

ip:172.30.1.16

state:active

Which is the same on NAS 1 where it is working. And as I said: When I configure "send targets", it automatically shows three targets in the static targets config, with the correct names, so at least something must be working.

[rickardnobel]

And you are sure that there are no masking options at all? No IP or IQN or MAC or anything?

Since you do get connection it does imply that the network way is working, and the non-precense of the LUNs itself is typically caused by LUNs not allowed to be shown from the iSCSI side.

You have done a "rescan all" at the Storage Adapter view?

[tzimmermann]

I just went through all options on both systems side by side to make sure I didn't miss one setting, but nothing.

They differed only by the time settings (the second NAS not being able to aquire a new time, thus being off clock by an hour).

I corrected that and now both have the same settings down to LUN settings.

Still no luck, even with a re-scan all.

[tzimmermann]

If there is a mask LUN option somewhere, it must not be on the WebGUI of the NAS, because I did not set anything like that in either of the devices.

The only option there is is username and password for the iSCSI LUNs, which is correct as I checked several times (and it works, at least from a Windows Host, AND the NAS says the connection is active)

[tzimmermann]

One additional thing:

I just noticed that the LUNs show up in the "Path" view, but not in the "Devices" view.

Sorry I am using all in German, so am translating:

Connected destinations: 16 (thats 3 more than without NAS 2, so good!)

Devices: 8 (same amount, so, no extra device)

Path: 16 (again, three more than what it used to be without NAS 2)

The last three entries are from NAS 2, the first you see is the last from NAS 1.

[rickardnobel]

The LUNs itself, they are not incredible small or large either?

[tzimmermann]

I posted a screenshot above this, sorry.

The LUNs are 1.9 TB, 1.9 TB and 1.6 TB (which is exactly the same as on the other NAS), and 2 TB LUNs are working fine from our SAN, too!

[rickardnobel]

Thomas Zimmermann wrote:

I just noticed that the LUNs show up in the "Path" view, but not in the "Devices" view.

Sorry I am using all in German, so am translating:

Keine problem, Ich verstehe ein bissen deutche. But not much, many years since school.

Strange situation you have, where the path, but not the LUN are available.

How is your network setup at the host? How many vmnics, vSwitches and vmkernel interface for iSCSI? Could you post a screenshot of the Networking configuration?

[tzimmermann]

Cool 😉

Vielen Dank für Ihre Hilfe!

btw. I found a way to force the client to use the english GUI. So here goes the config screenshots.

Our iSCSI-Network has two physical switches, every host has one NIC attached to each. On each switch one of our SAN Nodes is attached to have redundancy. (We actually bought the whole setup form a Storage supplier).

Since, for path redundancy, the SAN uses two different IP-Ranges to access the same LUNs, every host has to vmknics for each Subnet (172.30.1.0/24 and 172.30.21.0/24), and two pNICs to connect to either one of the pSwitches.

Then we have one standard vSwitch, the two pNICs attached to it, and the two vNICs attached to it.

Our NAS Systems are not for HA purposes, so they have just one NIC inside the 172.30.1.0/24 range, 172.30.1.200 in case of the working NAS 1, and 172.30.1.201 in case of the not working NAS 2.

This host has the IP address 16, so 10.31.140.16 /24 for Management, 172.30.1.16 and 172.30.21.16 for Storage.

[rickardnobel]

Du bist wilkommen!

On your vSwitch1, have you done any settings on the vmk1 and vmk2, like using active/unused directly to the nic ports or are the settings at default?

The same with the "networking" configuration on the Software iSCSI adapter (on Storage adapter view), are any vmk adapters bound to the iSCSI initiator?

[tzimmermann]

The Storage vmknics inherit their settings all from the vSwitch1, and for the switch both pNICs are active.

There are no port bindings configured for the iSCSI Initiator.

[rickardnobel]

So that should rule out any multipath mis configuration issues. And no jumbo frames enabled on the ESXi and  / or NAS device?

[tzimmermann]

Nope, jumboframes are deactivated throughout the whole chain: Neither SAN, NAS 1 or NAS 2 nor Switches nor pNICs or vmknics are configured to use them.

[rickardnobel]

Thomas Zimmermann wrote:

Nope, jumboframes are deactivated throughout the whole chain:

Nothing strange anywhere.

Do you have access to the shell on the ESXi host? (Direct console or SSH)

Could you run:

esxcli network ip connection list | grep 3260

[tzimmermann]

I get a long list of all connected iSCSI targets, uncluding eventual connections to vmdks on a certain datastore.

The ones that interest us would be the following:

~ # esxcli network ip connection list | grep 3260

tcp         0       0  172.30.1.16:53293               172.30.1.201:3260   ESTABLISHED      8775  vmkiscsid

tcp         0       0  172.30.1.16:62545               172.30.1.201:3260   ESTABLISHED      8775  vmkiscsid

tcp         0       0  172.30.1.16:59041               172.30.1.201:3260   ESTABLISHED      8775  vmkiscsid

[rickardnobel]

That looks good too of course. So the TCP sessions has been setup and is working as well.

It seems like all connectivity up to layer four is working fine and the network infrastructure is not complicated in your case. There are no obvious reason why the LUNs are not visible for VMFS formatting. I will give it another thought and see if I can come up with something.

[tzimmermann]

your time and expertise is most apreciated!