ESXI 5 - 2 Hosts and the best way to setup vSwitch's and Nic's for Redundancy

[TrevorDustan]

Hi All

Could anyone help me work out the best way to setup the vNetwork on 2 ESXI 5 host for redundancy.  Once i have managed to setup 2 hosts correctly i will be looking to use the same setup process for 6 hosts.  3 sites with 2 hosts at each site all managed from vCentre Server

I have 2 DL380 G7 Servers with ESXI 5 installed on a class 10, 8GB SD Card, i am looking to install VSA across the 2 hosts using 4TB of internal storage on each server (8*600GB 10k SAS).  Each server has a 4 port on-board gigabit NIC and I have installed a 2nd 4 port PCIe gigabit NIC, I have also x2 16 port switches with layer 3 routing.  I will be using vSphere 5 Standard Acceleration Kit so i am looking to use vCentre Server for management, vMotion for maintenance, HA for failover and 10/15 VM's (vCentre Server Std, SQL DB for vCentre Server, Exchange 2010, SQL Server 2008 R2, IIS Intranet, Helpdesk, Domain Controller 1, Domain Controller 2, AV Server, WSUS Server, SCCM Server and Terminal Server).

What would be the best way to setup and configure the network for performance and redundancy and am i missing anything?

my thoughts are, Teaming:-

vCenter - vswitch0 - port1 on NIC1 and port1 on NIC 2  -   port1 on NIC1 to physical switch 1    -    port1 on NIC2 to physical switch 2

vMotion - vswitch1 - port2 on NIC1 and port2on NIC 2  -   port2 on NIC1 to physical switch 1    -    port2 on NIC2 to physical switch 2

HA - vswitch3 - port3 on NIC1 and port3on NIC 2  -   port3 on NIC1 to physical switch 1    -    port3 on NIC2 to physical switch 2

VM's - vswitch4 - port4 on NIC1 and port4on NIC 2  -   port4 on NIC1 to physical switch 1    -    port4 on NIC2 to physical switch 2

or do i need an additional NIC in each server to spilt the 12 VM's to 6 VM's for every 2 ports over 2 NIC's, or maybe something else i've missed?

Thanks

[jamesbowling · Accepted solution]

In your case, to keep it simple and from what I can tell here is what my recommendation would be:

3 Standard vSwitches

vSwitch0:

• Management - vmnic0, vmnic2

vSwitch1:

• vMotion - vmnic1, vmnic3

vSwitch2:

• VM Network - vmnic4, vmnic5, vmnic6, vmnic7

The only reason I didn't say to split the VM Network into the other on-board adapters is because the difference of adapter types from the DL380 on-board and the PCIe quad.

[jamesbowling]

Also, you can see that you would just alternate cabling between your two switches.  If you absolutely don't want to have your VM Network rely on a single quad port card and to avoid losing that network when or if that card ever dies, we can change the vmnic config and change what NICs are attached to what vSwitches.

[GaneshNetworks]

Not sure whether I am directing you into a correct direction. Because I am not good enough in designing part. So, its all up to your consideration on my comment.

As of my knowledge, 4 ports on a single ESX are more than enough to setup  your vNetwork for redundancy. But here I wanna mention one thing that I am feeling about the port assignments. While you are creating vSwitch0 and vSwitch4, Map those two switches with the port 1 and 2. As like as while you creating vSwitch1 and vSwitch2, Map those switches with the ports 3 and 4.

By doing this, you can attain port level redundancy. Hope you consider this comment helpful.

~Ganesh

[TrevorDustan]

Hi jamesbowling, GaneshNetworks thanks for your suggestions.  Hopefully i've understood this correctly from both of your points and this is what i have come up with, would you say this is a good way to go.

Onboard NIC: vmnic0 vmnic1 vmnic2 vmnic3             HP Quad NIC: vmnic4 vmnic5 vmnic6 vmnic7         pSwitch: HP V1910-16 Port Gigabit Switch

vSwitch0:

• Management - vmnic0 into pSwitch1,       (standby) vmnic4 into pSwitch2    (I assume Management is vCenter Server?)
  

vSwitch1:

• vMotion - vmnic1 into pSwitch1,       (standby) vmnic5 into pSwitch2
  

vSwitch2:

• VM Network - vmnic2 + vmnic6 into pSwitch1,       (standby) vmnic3 + vmnic7 into pSwitch2
  

Do i need seperate or any ports for HA?

Many Thanks

Trevor

sorry had to edit just noticed a typo on the numbering of the vmnic's

[jamesbowling]

No. There is no need as HA is done through the management interface by vCenter.

James Bowling

Sent from my iPhone

[griffinboy]

What license type are you using? Only asking because you could make use of the distributed virtual switch to avoid management overheads.

In order to achieve HA on our clusters I pretty much always use the same formula:

• vSwitch0 - vmnic0 - management
• vSwitch1 - vmnic1 & 2 - iSCSI storage (2x iSCSI vmks where vnkx is bound to vmnic1 with vmnic2 un-used, and vmky bound to vmnic2 with vmnic1 unused) - also got multipath set on the iSCSI software adapter - vmware round robin.
• DVS - vmnic3, 4 & 5 - with at least one management port group on there to ensure management redundancy. Then I have other port groups like vMotion, fault tolerance, SAN, WAN, etc

all VMKs have their own unique VLAN apart from the iSCSI & SAN ones that share the same one.

We also make sure that the NICs are spread across multiple physical switches in case we lose one. i.e. if we lose a switch then we'd still have at least one vmnic working on vSwitch1 and the DVS.

Hope this helps in some way.

By the way, if anyone thinks we're doing it the wrong way too then please feel free to shout at me 🙂

[TrevorDustan]

Hi Griffinboy

We don’t have a license type yet as I’m in testing faze to see what works best, I’m thinking of ESXI 5 Std Acc Kit as we only have 3 sites (site1, 50 users - site2, 50 users - site3, 20 users) and I’m thinking of having 2 hosts at each site with a single 6 core cpu on each server and 30GB RAM ish, hosting 10 VM's at site1 but the same hardware setup at site2 as this will be the DR site where i will sync/backup the VM's to and site1 & site2 both have 10mb/10mb leased lines with site to site ipsec vpn.  I have just brought x2 DL380 G7, with a Class 10, 8GB 35/MB SD Card for ESXI install, I have also brought 8* 600GB 10k SAS and 8*1TB MDL 7.2k SAS drives for testing, if the 1TB MDL SAS are to slow they will be good enough for site3 which will have 2 hosts and 2 VM's (DC1 & DC2) for 20 users.

I'm new to vmware but from what i understand we won’t be using iSCSI for storage as we will be using the inbuilt drives within the server using HP P4000 VSA across the 2 hosts, however i am going to try and setup iSCSI for our backup this will be to a TS-QNAP 459-Pro II 8TB which has site to site or QNAP to QNAP block level real-time replication.

As for the vmnic's i don’t really understand what you've said "really sorry" i.e i got a bit confused when you said DVS and port groups? but does this mean there is only 1 port for management vmnic0 on vswitch0? but what happens if the physical nic to vmnic0 fails how do i gain access to manage or change nic settings.

Can i ask a bit more about VLANs and why the VM's are on separate VLAN? i have to re-configure the whole network across all 3 sites anyway so I have a choice, cant i just use different subnetting  i.e.

Site 1 - 192.168.1.X (255.255.0.0)

Site 2 - 192.168.2.X (255.255.0.0)

Site 3 - 192.168.3.X (255.255.0.0)

ESXI01 - 192.168.255.X (255.255.0.0) site1

ESXI02 - 192.168.255.X (255.255.0.0) site1

ESXI03 - 192.168.255.X (255.255.0.0) site2

ESXI04 - 192.168.255.X (255.255.0.0) site2

ESXI05 - 192.168.255.X (255.255.0.0) site3

ESXI06 - 192.168.255.X (255.255.0.0) site3

Thanks

Trevor

[griffinboy]

As for the vmnic's i don’t really understand what you've said "really sorry" i.e i got a bit confused when you said DVS and port groups? but does this mean there is only 1 port for management vmnic0 on vswitch0? but what happens if the physical nic to vmnic0 fails how do i gain access to manage or change nic settings.

DVS = Distributed Virtual Switch (only available with Enterprise Plus edition of vSphere) -> http://www.vmware.com/products/vnetwork-distributed-switch/overview.html

In order to understand what a port group is please refer to page 6 of this document: http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Anyway, I only use one physical NIC for my vSwitch0 because I have another Management network (port group) on my DVS and therefore if vmnic0 goes pop I still have access to the management network via the DVS.

Can i ask a bit more about VLANs and why the VM's are on separate VLAN? i have to re-configure the whole network across all 3 sites anyway so I have a choice, cant i just use different subnetting  i.e.

Site 1 - 192.168.1.X (255.255.0.0)

Site 2 - 192.168.2.X (255.255.0.0)

Site 3 - 192.168.3.X (255.255.0.0)

ESXI01 - 192.168.255.X (255.255.0.0) site1

ESXI02 - 192.168.255.X (255.255.0.0) site1

ESXI03 - 192.168.255.X (255.255.0.0) site2

ESXI04 - 192.168.255.X (255.255.0.0) site2

ESXI05 - 192.168.255.X (255.255.0.0) site3

ESXI06 - 192.168.255.X (255.255.0.0) site3

The use of VLANs is completely up to you and your design requirements.

We use VLANs extensively for added layer 2 security.

You could have 2 customers or departments and you don't want these guys to be able to communicate with each other. Then you can create one port group each, give each port group a distinct VLAN (that needs configuring on your upstream physical switches too), dump the VMs in their respective port group by assigning the vNIC to the respective port group and voila. The VMs will be completely isolated from each other. I'm no networking guru but as far as I know this is a security best practice.

I'm not sure what your VMs functions are but you could have one VLAN for Internet, one for your LAN, one for your SAN network (if you had one), we also have distinct VLANs and subnets for vMotion or Fault Tolerance, etc...

Then if you have a VM that needs to talk to both the SAn and access the Internet then you give it two vNICs, one  in the Internet port group, the other in the SAN port group and that's it.

Whatever you do with VLANs, I recommend you use the same VLAN IDs across all your sites, otherwise it can be a bit of a headache trying to figure out which is which...