tasonis
Contributor
Contributor

Network Infrastructure using Dell PC6248s and 6 NICs

We have 3 Dell Servers for ESX hosts (2x 2950s, 1x 2850) with an Equallogic PS100E iscsi running between two Dell PowerConnect 6248s. We also have two 2650's, one to be used for virtual center and the other as a backup proxy. The ESX hosts all have 6 NICs, the vcms and backup have two NICs. What would be the recommended setup with this infrastructure, and how should I setup VLANs on the 6248s? Setting up the 6248s has proven to be quite the task and we have had issues routing through the VLANs. Thanks for any reccomendations!

0 Kudos
8 Replies
Tricue
Contributor
Contributor

I can help you out with some ideas. We utilize 4-6 NICs and VLAN trunking on each of our ESX 3 servers in our VI cluster.

First thing to do is to setup your physical switch ports for VLAN trunking. We use CISCO Catalyst 3750 switches.

Create you VLANs if you don't already have them. For example:

vlan 100

name INSIDE

!

vlan 800

name DMZ

!

vlan 2541

name VMOTION

!

Create a port channel interface for each ESX 3 host. In this example I have two ESX 3 servers.

interface Port-channel10

description >> ESX0, Chaqnnel-Group 10, VLANs 1,100,800,2541 <<

switchport access vlan 100

switchport trunk encaptulation dot1q

switchport trunk allowed vlan 1,100,800,2541

switchport mode trunk

!

interface Port-channel11

description >> ESX1, Chaqnnel-Group 11, VLANs 1,100,800,2541 <<

switchport access vlan 100

switchport trunk encaptulation dot1q

switchport trunk allowed vlan 1,100,800,2541

switchport mode trunk

!

Configure the 6 GigabitEthernet ports on the switch for each ESX host as members of their respective channel groups. In this example I am using Gi1/0/1-6 for ESX0 and Gi2/0/1-6 for ESX1.

interface GigabitEthernet1/0/1

description >> ESX0, Channel-Group 10 <<

switchport access vlan 100

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,100,800,2541

switchport mode trunk

channel-group 10 mode on

spanning-tree portfast

!

interface GigabitEthernet1/0/2

description >> ESX0, Channel-Group 10 <<

switchport access vlan 100

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,100,800,2541

switchport mode trunk

channel-group 10 mode on

spanning-tree portfast

!

interface GigabitEthernet1/0/3

description >> ESX0, Channel-Group 10 <<

switchport access vlan 100

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,100,800,2541

switchport mode trunk

channel-group 10 mode on

spanning-tree portfast

!

And so forth... For ESX1:

interface GigabitEthernet2/0/1

description >> ESX0, Channel-Group 11 <<

switchport access vlan 100

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,100,800,2541

switchport mode trunk

channel-group 11 mode on

spanning-tree portfast

!

And so forth...

Now you are ready to physically connect each NIC to it's designated switch port, and to install/configure the ESX servers.

If you are installing ESX from scratch, at this point you can enter the VLAN ID (100) for the Service Console during the installation. If you already have ESX installed you will have to make the change from the system console because you will no longer have access to the Service Console until it has the right VLAN ID. To change the Service Console from the system console:

Log in as root.

type:

esxcfg-vswitch vswitch0 -p "Service Console" -v 100

You should now be able to access the Service Console from VLAN 100. Do the same for ESX1.

Now connect to the ESX server with VI Client or VI Center and configure your vSwitch and NIC teaming.

In vSwitch0 Properties go to the Network Adapters tab and add each of your NICs. Then go back to the Ports tab and edit the vSwitch. Adjust the number of Ports if you wish and then go to the "NIC Teaming" tab. Configure teaming as follows:

Load Balancing: Route based on ip hash

Network Failover Detection: Link Status only

Notify Switches: Yes

Rolling Failover: No

Click ok Once it finished updating ESX, you are ready to add your DMZ VM Port Group (vlan 800) and VMotion Port (vlan 2541). Do the same for each ESX server. When you are done you will have something that looks a little like this (click on the image):

!vlantrunking.jpg!

Now you have load balancing, redundancy, and logical separation of network segments. You can lose up to 5 NICs and still have all services available (VM Port Groups, VMotion, and Service Console).

We run this configuration in our production environment and it works great. Never had a problem. It is also very useful for 1U ESX servers that have two onboard NICs and one PCI slot that is used for the Fibre HBA. This way you can have NIC redundancy and still have multiple networks and ports (VMotion, Service Console, etc...).

Hope this helps.

Tricue.

tasonis
Contributor
Contributor

Wow! Thanks for the fast reply, lots of great information we'll be using from your post.

0 Kudos
WillemB
Enthusiast
Enthusiast

A variation on the reply you've already gotten. We use the 6 nics as follows.

1. Service console

2. VMotion

3. User LAN (trunk)

4. User LAN2 (trunk)

5. Storage LAN1 (trunk)

6. Storage LAN2 (trunk)

(7.) Remote control (ILO/RILO)

Then set up the cisco switches to trunk with 3,4,5,6 and set up 1,2,7 in their dedicated VLAN's. I'm no cisco expert so i can't give the cool dump the other guy gave ;-).

tasonis
Contributor
Contributor

Well, here is what we decided on to simplify setup...Thanks Everyone.

1441_1441.jpg

0 Kudos
Tricue
Contributor
Contributor

Awesome. That's the way to do it.

0 Kudos
Cody_Page
Enthusiast
Enthusiast

Hi Tasonis,

We are using an identical setup. Same arrarys, hosts and switches. Can you give me a dump of the commands you issued your Dell switches for this setup?

Thanks

Cody

0 Kudos
tasonis
Contributor
Contributor

Here you go!

!Current Configuration:

!System Description "Dell 48 Port Gigabit Ethernet, 2.0.0.12, VxWorks5.5.1"

configure

vlan database

vlan 7-8,200

exit

snmp-server location "Winter Haven FL"

snmp-server contact "John Gunter"

hostname "ESX Server SAN Switches"

sntp unicast client enable

sntp server 192.168.9.5

clock timezone 7 minutes 0 zone "EST"

stack

member 1 2

member 2 2

exit

ip address 10.10.10.10 255.255.255.0

ip default-gateway 10.10.10.1

ip address vlan 999

ip routing

ip route 0.0.0.0 0.0.0.0 192.168.9.253

interface vlan 1

routing

ip address 192.168.9.180 255.255.255.0

exit

interface vlan 8

routing

ip address 192.168.8.1 255.255.255.0

exit

username "admin" password bca92e811d9a7e7e2b6864659df17630 level 15 encrypted

flowcontrol

!

interface ethernet 1/g1

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g2

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g3

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g4

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g5

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g6

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g7

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g8

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g9

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g10

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g11

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g12

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 1/g13

channel-group 1 mode on

mtu 9000

exit

!

interface ethernet 1/g14

channel-group 1 mode on

mtu 9000

exit

!

interface ethernet 1/g15

channel-group 1 mode on

mtu 9000

exit

!

interface ethernet 1/g16

channel-group 1 mode on

mtu 9000

exit

!

interface ethernet 1/g17

channel-group 2 mode on

mtu 9000

exit

!

interface ethernet 1/g18

channel-group 2 mode on

mtu 9000

exit

!

interface ethernet 1/g19

channel-group 3 mode on

mtu 9000

exit

!

interface ethernet 1/g20

channel-group 3 mode on

mtu 9000

exit

!

interface ethernet 1/g21

channel-group 3 mode on

mtu 9000

exit

!

interface ethernet 1/g22

channel-group 3 mode on

mtu 9000

exit

!

interface ethernet 1/g23

channel-group 4 mode on

mtu 9000

exit

!

interface ethernet 1/g24

channel-group 4 mode on

mtu 9000

exit

!

interface ethernet 2/g1

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g2

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g3

mtu 9000

switchport access vlan 8

exit

interface ethernet 2/g4

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g5

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g6

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g7

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g8

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g9

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g10

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g11

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g12

mtu 9000

switchport access vlan 8

exit

!

interface ethernet 2/g13

channel-group 1 mode on

mtu 9000

exit

!

interface ethernet 2/g14

channel-group 1 mode on

mtu 9000

exit

!

interface ethernet 2/g15

channel-group 2 mode on

mtu 9000

exit

!

interface ethernet 2/g16

channel-group 2 mode on

mtu 9000

exit

!

interface ethernet 2/g17

channel-group 2 mode on

mtu 9000

exit

!

interface ethernet 2/g18

channel-group 2 mode on

mtu 9000

exit

!

interface ethernet 2/g19

channel-group 3 mode on

mtu 9000

exit

!

interface ethernet 2/g20

channel-group 3 mode on

mtu 9000

exit

!

interface ethernet 2/g21

channel-group 4 mode on

mtu 9000

exit

!

interface ethernet 2/g22

channel-group 4 mode on

mtu 9000

exit

!

interface ethernet 2/g23

channel-group 4 mode on

mtu 9000

exit

!

interface ethernet 2/g24

channel-group 4 mode on

mtu 9000

exit

!

interface port-channel 1

switchport mode general

no switchport general acceptable-frame-type tagged-only

switchport general allowed vlan add 7-8,200 tagged

exit

!

interface port-channel 2

switchport mode general

no switchport general acceptable-frame-type tagged-only

switchport general allowed vlan add 7-8,200 tagged

exit

!

interface port-channel 3

switchport mode general

no switchport general acceptable-frame-type tagged-only

switchport general allowed vlan add 7-8,200 tagged

exit

!

interface port-channel 4

switchport mode general

no switchport general acceptable-frame-type tagged-only

switchport general allowed vlan add 7-8,200 tagged

exit

snmp-server community public ro

exit

0 Kudos
Cody_Page
Enthusiast
Enthusiast

Thanks!

I also noticed you're not using Spanning tree portfast on your ESX host ports. Is there any specific reason for this?

Thanks

Cody

0 Kudos