From FC to iSCSI SAN using ESXi 5.1

[stevehoot]

Hi all,

We have 4 hosts with ESX5.1 installed and use vCentre to manage with Enterprise licences. All hosts started life as 3.5 and we've upgraded ever since. Due to be replaced in 6 months.

One key thing is that the VMKernel port is a dedicated management network using two physical NIC's for redundancy.

Servers all used FC HBA's all multihomed over a mesh FC fabric. We are replacing the FC SAN with an iSCSI only SAN.

My question is what is the best way to implement this? I've created a vSwitch with two VMKernel ports on it. The vSwitch is fed into two seperate NIC's set as Active for failover. This is all fine, and done on all of the hosts.

However I was planning a dedicated iSCSI vLAN with just the SAN's on it. This appears to be a problem, as I cannot set a particular default gateway on the port groups as they are VMKernel ports and will only use the default gateway I have set on the managment network which won't work!

vSwitch0

Management

vmk1:192.168.200.102

vSwitch6

iSCSI2

vmk3: 192.168.30.162

iSCSI

vmk2: 192.168.30.161

Is there only a single routing table shared for all VMKernel ports? If so how should I implement this? I'd rather not have iSCSI on my management network but following the guides online I get this far and cannot route iSCSI interfaces I've setup outside of it's own subnet.... Is this by design?

Any advice much apprecaited!

Thanks - Steve

[stevehoot]

Bit of clarrification / insight:

vSwitch0

Management

vmk1:192.168.200.102

DG is 192.168.200.250.

vSwitch6

iSCSI2

vmk3: 192.168.30.162

iSCSI

vmk2: 192.168.30.161

If I try to add a gateway to the two new port groups above I get taken to the UI for the VMK1 port group resulting in modifying the management gateway.

[rickardnobel]

Where are your iSCSI storage located? It should really only be on the local subnet (VLAN) defined for storage traffic and not be routed.

If the iSCSI SAN is on the same IP network then you do not need any specific default gateway for IP storage, which you have noted is also not possible.

[stevehoot]

Thanks for the quick reply.

At the moment it's not anywhere! We've brought the iSCSI SAN's but I've yet to deploy it - just designing it at the moment. Thinking of giving it it's own little vLAN with high QoS on it across the backbone.So it would be on it's own subnet with nothing else in the way.

I suppose I don't actually NEED it to be routable... more of something that I thought would be useful as the storage network will need to be routeable from the management network anyway, seems odd there's no option to allow it.

Just to be 100% sure, what I appear to have done is correct, and that ESXI doesn't allow an iSCSI VMK to have a default gateway on it?

Thanks again.

[rickardnobel · Accepted solution]

The very recommended best practice is to configure a separate VLAN with IP network used only for iSCSI, possible even with dedicated layer two switches used only for iSCSI, depending on the type and quality of the switches.

stevehoot wrote:

Just to be 100% sure, what I appear to have done is correct, and that ESXI doesn't allow an iSCSI VMK to have a default gateway on it?

It is not really that the storage network can not have a default gateway, but instead that the ESXi operating system (vmkernel) has IP stack with internal routing table shared over the different functions like management, vMotion, storage and other.

This means that you have a common default gateway per host and not per function, and this will also most typically mean that the "common" default gateway will be on the management network.