We have 3 Dell Servers for ESX hosts (2x 2950s, 1x 2850) with an Equallogic PS100E iscsi running between two Dell PowerConnect 6248s. We also have two 2650's, one to be used for virtual center and the other as a backup proxy. The ESX hosts all have 6 NICs, the vcms and backup have two NICs. What would be the recommended setup with this infrastructure, and how should I setup VLANs on the 6248s? Setting up the 6248s has proven to be quite the task and we have had issues routing through the VLANs. Thanks for any reccomendations!
I can help you out with some ideas. We utilize 4-6 NICs and VLAN trunking on each of our ESX 3 servers in our VI cluster.
First thing to do is to setup your physical switch ports for VLAN trunking. We use CISCO Catalyst 3750 switches.
Create you VLANs if you don't already have them. For example:
vlan 100
name INSIDE
!
vlan 800
name DMZ
!
vlan 2541
name VMOTION
!
Create a port channel interface for each ESX 3 host. In this example I have two ESX 3 servers.
interface Port-channel10
description >> ESX0, Chaqnnel-Group 10, VLANs 1,100,800,2541 <<
switchport access vlan 100
switchport trunk encaptulation dot1q
switchport trunk allowed vlan 1,100,800,2541
switchport mode trunk
!
interface Port-channel11
description >> ESX1, Chaqnnel-Group 11, VLANs 1,100,800,2541 <<
switchport access vlan 100
switchport trunk encaptulation dot1q
switchport trunk allowed vlan 1,100,800,2541
switchport mode trunk
!
Configure the 6 GigabitEthernet ports on the switch for each ESX host as members of their respective channel groups. In this example I am using Gi1/0/1-6 for ESX0 and Gi2/0/1-6 for ESX1.
interface GigabitEthernet1/0/1
description >> ESX0, Channel-Group 10 <<
switchport access vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,100,800,2541
switchport mode trunk
channel-group 10 mode on
spanning-tree portfast
!
interface GigabitEthernet1/0/2
description >> ESX0, Channel-Group 10 <<
switchport access vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,100,800,2541
switchport mode trunk
channel-group 10 mode on
spanning-tree portfast
!
interface GigabitEthernet1/0/3
description >> ESX0, Channel-Group 10 <<
switchport access vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,100,800,2541
switchport mode trunk
channel-group 10 mode on
spanning-tree portfast
!
And so forth... For ESX1:
interface GigabitEthernet2/0/1
description >> ESX0, Channel-Group 11 <<
switchport access vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,100,800,2541
switchport mode trunk
channel-group 11 mode on
spanning-tree portfast
!
And so forth...
Now you are ready to physically connect each NIC to it's designated switch port, and to install/configure the ESX servers.
If you are installing ESX from scratch, at this point you can enter the VLAN ID (100) for the Service Console during the installation. If you already have ESX installed you will have to make the change from the system console because you will no longer have access to the Service Console until it has the right VLAN ID. To change the Service Console from the system console:
Log in as root.
type:
esxcfg-vswitch vswitch0 -p "Service Console" -v 100
You should now be able to access the Service Console from VLAN 100. Do the same for ESX1.
Now connect to the ESX server with VI Client or VI Center and configure your vSwitch and NIC teaming.
In vSwitch0 Properties go to the Network Adapters tab and add each of your NICs. Then go back to the Ports tab and edit the vSwitch. Adjust the number of Ports if you wish and then go to the "NIC Teaming" tab. Configure teaming as follows:
Load Balancing: Route based on ip hash
Network Failover Detection: Link Status only
Notify Switches: Yes
Rolling Failover: No
Click ok Once it finished updating ESX, you are ready to add your DMZ VM Port Group (vlan 800) and VMotion Port (vlan 2541). Do the same for each ESX server. When you are done you will have something that looks a little like this (click on the image):
!vlantrunking.jpg!
Now you have load balancing, redundancy, and logical separation of network segments. You can lose up to 5 NICs and still have all services available (VM Port Groups, VMotion, and Service Console).
We run this configuration in our production environment and it works great. Never had a problem. It is also very useful for 1U ESX servers that have two onboard NICs and one PCI slot that is used for the Fibre HBA. This way you can have NIC redundancy and still have multiple networks and ports (VMotion, Service Console, etc...).
Hope this helps.
Tricue.
Wow! Thanks for the fast reply, lots of great information we'll be using from your post.
A variation on the reply you've already gotten. We use the 6 nics as follows.
1. Service console
2. VMotion
3. User LAN (trunk)
4. User LAN2 (trunk)
5. Storage LAN1 (trunk)
6. Storage LAN2 (trunk)
(7.) Remote control (ILO/RILO)
Then set up the cisco switches to trunk with 3,4,5,6 and set up 1,2,7 in their dedicated VLAN's. I'm no cisco expert so i can't give the cool dump the other guy gave ;-).
Well, here is what we decided on to simplify setup...Thanks Everyone.
Awesome. That's the way to do it.
Hi Tasonis,
We are using an identical setup. Same arrarys, hosts and switches. Can you give me a dump of the commands you issued your Dell switches for this setup?
Thanks
Cody
Here you go!
!Current Configuration:
!System Description "Dell 48 Port Gigabit Ethernet, 2.0.0.12, VxWorks5.5.1"
configure
vlan database
vlan 7-8,200
exit
snmp-server location "Winter Haven FL"
snmp-server contact "John Gunter"
hostname "ESX Server SAN Switches"
sntp unicast client enable
sntp server 192.168.9.5
clock timezone 7 minutes 0 zone "EST"
stack
member 1 2
member 2 2
exit
ip address 10.10.10.10 255.255.255.0
ip default-gateway 10.10.10.1
ip address vlan 999
ip routing
ip route 0.0.0.0 0.0.0.0 192.168.9.253
interface vlan 1
routing
ip address 192.168.9.180 255.255.255.0
exit
interface vlan 8
routing
ip address 192.168.8.1 255.255.255.0
exit
username "admin" password bca92e811d9a7e7e2b6864659df17630 level 15 encrypted
flowcontrol
!
interface ethernet 1/g1
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g2
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g3
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g4
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g5
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g6
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g7
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g8
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g9
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g10
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g11
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g12
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 1/g13
channel-group 1 mode on
mtu 9000
exit
!
interface ethernet 1/g14
channel-group 1 mode on
mtu 9000
exit
!
interface ethernet 1/g15
channel-group 1 mode on
mtu 9000
exit
!
interface ethernet 1/g16
channel-group 1 mode on
mtu 9000
exit
!
interface ethernet 1/g17
channel-group 2 mode on
mtu 9000
exit
!
interface ethernet 1/g18
channel-group 2 mode on
mtu 9000
exit
!
interface ethernet 1/g19
channel-group 3 mode on
mtu 9000
exit
!
interface ethernet 1/g20
channel-group 3 mode on
mtu 9000
exit
!
interface ethernet 1/g21
channel-group 3 mode on
mtu 9000
exit
!
interface ethernet 1/g22
channel-group 3 mode on
mtu 9000
exit
!
interface ethernet 1/g23
channel-group 4 mode on
mtu 9000
exit
!
interface ethernet 1/g24
channel-group 4 mode on
mtu 9000
exit
!
interface ethernet 2/g1
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g2
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g3
mtu 9000
switchport access vlan 8
exit
interface ethernet 2/g4
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g5
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g6
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g7
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g8
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g9
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g10
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g11
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g12
mtu 9000
switchport access vlan 8
exit
!
interface ethernet 2/g13
channel-group 1 mode on
mtu 9000
exit
!
interface ethernet 2/g14
channel-group 1 mode on
mtu 9000
exit
!
interface ethernet 2/g15
channel-group 2 mode on
mtu 9000
exit
!
interface ethernet 2/g16
channel-group 2 mode on
mtu 9000
exit
!
interface ethernet 2/g17
channel-group 2 mode on
mtu 9000
exit
!
interface ethernet 2/g18
channel-group 2 mode on
mtu 9000
exit
!
interface ethernet 2/g19
channel-group 3 mode on
mtu 9000
exit
!
interface ethernet 2/g20
channel-group 3 mode on
mtu 9000
exit
!
interface ethernet 2/g21
channel-group 4 mode on
mtu 9000
exit
!
interface ethernet 2/g22
channel-group 4 mode on
mtu 9000
exit
!
interface ethernet 2/g23
channel-group 4 mode on
mtu 9000
exit
!
interface ethernet 2/g24
channel-group 4 mode on
mtu 9000
exit
!
interface port-channel 1
switchport mode general
no switchport general acceptable-frame-type tagged-only
switchport general allowed vlan add 7-8,200 tagged
exit
!
interface port-channel 2
switchport mode general
no switchport general acceptable-frame-type tagged-only
switchport general allowed vlan add 7-8,200 tagged
exit
!
interface port-channel 3
switchport mode general
no switchport general acceptable-frame-type tagged-only
switchport general allowed vlan add 7-8,200 tagged
exit
!
interface port-channel 4
switchport mode general
no switchport general acceptable-frame-type tagged-only
switchport general allowed vlan add 7-8,200 tagged
exit
snmp-server community public ro
exit
Thanks!
I also noticed you're not using Spanning tree portfast on your ESX host ports. Is there any specific reason for this?
Thanks
Cody