ESX configuration for iSCSI failover - actual IP addresses

[dchana]

Hi everyone,

I've been surfing through the communities trying to find an answer for this question with no luck at all - I'm still not sure about the right ESX configuration. Let me explain it in detail:

We have several HP ProLiant servers with ESX 3.0.2 up & running, but no VMs configured yet (so we can reboot the servers as desired). Each of this servers have 3 physical Ethernet Gigabit NICs, two of wich will be used to configure a high availability iSCSI SAN with an EMC target. There are 2 switches and each of them have 2 Ethernet Gigabit connections to the target. This is gonna be a software-initiated iSCSI configuration.

Talking about the IP subnets, there is already configured a VLAN for the iSCSI SAN in the switches. This VLAN is 192.168.0.0 with netmask 255.255.255.192. And there is another subnet used for management purposes and connected to the enterprise network, which is 172.16.0.0/16.

So this is where things get interesting. I'll call the first ethernet card -used for management and connection to enterprise network- NIC0 and the other two -used for iSCSI- NIC1 and NIC2. What I would do considering only 1 ESX server:

1. Assign IP addresses for each NIC (on ESX server) and EMC's target:

• NIC0 = 172.16.0.35

• NIC1 = 192.168.0.1

• NIC2 = 192.168.0.2

• Target1 = 192.168.0.51

• Target2 = 192.168.0.52

• Target3 = 192.168.0.53

• Target4 = 192.168.0.54

2. Create a Virtual Switch (vSwitch0) and assign the NIC0. This is the switch that VMs will use through a "Virtual Machine Port Group". In this vSwitch0 create a Service Console. Question: does this Service Console have to be on the same subnet as the physical NIC0? i.e. Service Console = 172.16.0.56?

3. Create a new Virtual Switch (vSwitch1) and assign both NIC1 and NIC2 (bonding) for failover. This vSwitch1 will be dedicated only to iSCSI network. I've read that for iSCSI to work properly there IS necessary to create both a Service Console and a VMkernel. So create those and assign to vSwitch1. Question: does this Service Console AND VMkernel have to be on the same subnet as the physical NIC1 and NIC2? i.e. Service Console 2 = 192.168.0.31 and VMkernel = 192.168.0.32?

4. Configure Firewall policies to allow iSCSI (if not done before). For failover configure "NIC teaming" properties on vSwitch, then rescan, add targets, rescan, assign LUNs, rescan over to infinity Maybe reboot server.

What do you think about it? Should it work this way? I really need to get over this or my brain is gonna freak out definitely Please post here your answers and guesses, I'll try this configuration on monday or tuesday and post here the results.

Regards!

[christianZ]

Maybe that can help:

http://communities.vmware.com/thread/101956

[dkfbp]

Hi,

1. You have one thing wrong here. You don't define ip addresses on the NICS. on vswitch0 you give the ip address to the service console and on vswitch1 you define the ip addresses on service console and vmkernel.

2. Read number 1. You only define one ip address. You can just use the one you have planned for NIC0.

3. Read number 1. You don't define ip addresses on NICS. You define IP addresses for service console and vmkernel. They have to be on the iSCSI network.

4. Correct.

I have a few things I would like to point out. What about vmotion? You need a vmkernel for vmotion and the only one you have is on the iSCSI network. iSCSI traffic should be seperated from vmotion. You could add a vmkernel to vswitch0 but then you would have SC, virtual machine traffic and vmotion on the same NIC. Not something I would do. I don't know what kind of servers you are using. But if possible you should add a dual nic to the server. My prefered iSCSI network config in a 5 pNIC scenario:

vswitch0 - Service Console - pNIC0

vswitch1 - Vmkernel for vmotion - pNIC1

vswitch2 - Network for virtual machines - pNIC2

vswitch3 - iSCSI network vmotion and service console - pNIC3 and 4

Frank

[dchana]

Hi Frank,

thanks for your very useful answer --- I'll try it this week and report here for everyone interested.

The servers will not have a very heavy network load, instead they'll be more "CPU burners". And you're right, I didn't think about VMotion. Maybe I could ask for another pNIC in the servers and use it for VMotion. But 5 pNICs seems impossible right now. So I think I will definitely try this setup if I get the fourth NIC:

vSwitch0 - Service Console and network for VMs - pNIC0
vSwitch1 - VMotion (with VMkernel)- pNIC1
vSwitch2 - iSCSI network (with Service Console and VMkernel) - pNIC2 and pNIC3

If I can't get the fourth NIC I just will have to "join" vSwitch0 and vSwitch1 together and cross my fingers.

I just have one more question. If you doesn't define IP addresses on the NICs and, let's say, you want a iSCSI failover with 4 pNICs, how ESX hadles this? Will you need more VMkernels on the same vSwitch or something?

Keep in touch ---

Best regards,

Daniel.

[dkfbp]

Hi,

I have tested iSCSI failover in my test enviroment. And this is what I discovered.

I have two pNIC in my iSCSI vswitch that contains of a service console and vmkernel. The iSCSI initiatior in ESX

does not use load balancing so it will only use one pNIC at a time. I have tried pulling out the cable currently in use

in the physical switch and imediately I saw iSCSI traffic running on the other port.

Frank

[dkfbp · Accepted solution]

How did it work out for you Daniel?

Frank

[dchana]

Worked like a charm!

We just configured the ESX servers following the excellent guidelines from Frank --- and everything worked on first try! Although we just deployed a simpler architecture with no VMotion and only 3 NICs:

vSwitch0 (NIC0) --- Service Console + Virtual Machine Port Group
vSwitch1 (NIC1, NIC2) --- Service Console + VMkernel

And as Frank stated, we didn't assign IP addresses to the physical NICs, but to the virtual interfaces (Service Consoles and VMkernel).

Maybe next week we'll install another NIC and try VMotion.

Keep in touch.

Best regards,

Daniel.