Setting up my Network: Multiple Subnets, iSCSI, and HA isn't working.

[AAkins]

Greetings everyone.

So I am setting up my very first VMWare environment, using vSphere. The hardware configuration is as follows:

2xHosts:

HP DL360 G5 with 2xQuadCore 3.0GHz Xeons, 20 GB RAM, and 6 NICs

NIC 0: Connected to a cisco catalyst in our internal network, subnet 172.16.4.*

NIC 1: Connected to a vlan on a catalyst for VMotion traffic, subnet 172.16.6.*

NIC 2: Connected to a vlan on a catalyst for iSCSI traffic, subnet 172.16.5.*

NIC 3: Connected to a catalyst in our DMZ network, subnet 172.16.0.*

NIC 4: Connected to the same catalyst as NIC 0

NIC 5: Unused

Each host is running ESX 4.0. The following vSwitches are in place:

vSwitch 0 --> NIC 0, Service Console

vSwitch 1 --> NIC 1, vmKernel with VMotion enabled

vSwitch 2 --> NIC 2, vmKernel

vSwitch 3 --> NIC 3, VM network

vSwitch 4 --> NIC 4, VM network

The iSCSI network also has an EMC AX-4 on it, which the ESX machines get their datastores.

The vCenter server is a virtual Windows 2008 server connected to vSwitch 4.

Host 1

Service: 172.16.4.29

iSCSI: 172.16.5.21

vMotion: 172.16.6.21

Host 2

Service: 172.16.4.32

iSCSI: 172.16.5.23

vMotion: 172.16.6.23

Note that the iSCSI and VMotion subnets have no gateways - they only have the hosts/SANs on them.

For networking, the default gateway is 172.16.4.1.

Everything is up and running bascally - but HA doesn't want to work. I get an error: Could not reach Isolation Address 172.16.4.1.

Note that for the vmKernel, I have the default gateway as 172.16.6.1 - is this correct? I would think it wouldn't matter...

So... any ideas as to why HA isn't working? Is the network design flawed?

Thank you for any assistance you might have.

[mbaar]

Hello AAkins,

1. remove the Gateway from the vMotion Port and set an Def. Gateway only for the Serviceconsole

for ESX 3.x and 4.x all Def. Gateway must be reachable, for most Enviroment you dont need a def. gateway vor Kernelports

2. for a better and simple config, make one vswitch for all portgroups. The portgroup use vlan tags for separete the traffic. Connect all Nics to this vswitch and make an "MAC Hash based load balancing" and you have more bandwith and redundancy

3. alternativ make an 2. and 3. vswitch, connect on every vswitch one nic and configure one kernel port per switch for iscsi traffic. Use different ip-networks per switch and make a connect to SP1 and SP2 from the AX-150, then you can use multipathing for the SAN. Please check the AX Docu for this config, with an CX is this running fine.

regards marco

[Penic_Albin]

Is address 172.16.4.1 pingable from service console? Do you receive replay from it? This can be issue if your physical switch wont replay on ICMP.

One more think about networking. Best practice is that you have at least two NICs per vSwitch/network connection.

[AAkins]

Granted, I'm not a network guru, but I don't completely understand.

This environment is coming up in an existing physical network. The existing network (with physical machines) is a DMZ network (on its own catalyst 2960) 172.16.0.* and the internal network (on another catalyst 2960) 172.16.4.* . When we purchased the AX-4, we got a catalyst 3560, and made ports 2-4 be the 172.16.6.* subnet VLAN (for vmotion) and the rest of the ports the 172.16.5.* VLAN (for iSCSI). The AX-4's 4 data ports were plugged into the iSCSI VLAN on the 3560, and the service port was plugged into the internal switch.

So I guess I don't understand VMWare/Networking well enough to understand how I could make one vSwitch and put every NIC into it - which physical switch would these NICs be connected to, then?

I guess I could put vmnic0 and vmnic4, (service console and VM machines into the internal subnet) on the same vSwitch, and make two port groups on it - but every other port group is on a different Physical (not Trunk VLAN) subnet/switch - so don't those NICs have to be plugged into those physical ports?

I apologize for a lack of understanding, we're a small shop, and we don't have any big-time networking experience here. Simple cisco switches and physical networks is what we do.

Thanks for any help.

[AAkins]

Re: second suggestion.

You're right on the money - I logged into the service console and it cannot ping the gatway 172.16.4.1. So that is obviously a big part of the problem. I'm going to look into that.

As far as doubling up on NICs is concerned, you are correct. Right now I have 6 NICs connecting to 4 subnets, so not enough to double up on everything. I can add 4 more NICs, but I need to wait till I get budget approval to do that. I'd like to figure out the basic network topology first, then I can go to my boss and tell him for better performance/failover we need the extra 4xNIC cards.

Thanks.

[mbaar]

ok,

1. for separate physical switches (no bridging) use separate virtual switches

2. my recommends - use separate nics and switches for iSCSI, is this not so use min. vlans for iSCSI

In the ESX you have Portgroups connected to a Virtual Switch with Uplinks to the Physical Switch. The Uplinks to the Physical World can Loadbalanceing, Failover and Vlan Tagging.

Portgroup Virtual Switch Physical Switch

• Service Console -------->| vswitch0 | -------NIC0--------------------------> |internal Catalyst

• VMNetwork -------------->| vswitch0 | -------NIC4--------------------------> |internal Catalyst

• vMotion ----VLAN1-->| vswitch1 | -------NIC1-VLAN1+2 Tagged---> | SAN Catalyst

• iSCSI ---------VLAN2 ---> | vswitch1 | -------NIC2-VLAN1+2 Tagged---> | SAN Catalyst

• VMDMZNet -------------->| vswitch2 | -------NIC3----------------------------> | DMZ Catalyst

• | vswitch2 | -------NIC2----------------------------> | DMZ Catalyst

for the Uplink Nics use "MAC based Loadbalancing", this dont need any switchconfig, for vswitch1 use tagged ports on the Catalyst with vlan 12 menbership and the correct vlan for the Portgroup. For vswitch02 use no vlan's, no vlans for the Portgroups.

[Penic_Albin]

See attached file with network configuration.

I recommend that you and your network guys read this doc

If you find this information useful, please award points for "correct" or "helpful".