ESXi 5, Link Bundling, VLANs, and Management Interface

[jasonvp]

Greetings -

I suspect that the answer to my query is: "Buy another Intel NIC" but here goes:

I have an ESXi server here with 2 Intel GigE NICs, connected to the same upstream Cisco managed switch.  One NIC (vmnic0) is connected to VLAN 200 while the second (vmnic1) is connected to VLAN 300.  Ports on the Cisco are set to access mode.

Internally on the ESXi server, vmnic0 is connected to the "public" vSwitch, while vmnic1 is connected to the "private" vSwitch.

I also set the ESXi management IP to the same /24 that the private vSwitch.  This is key, I suspect.

I wanted to try and bundle the two GigE interfaces into a single 2xGigE connection, and trunk both VLANs 200 and 300 across it.  After struggling through the menu-driven UI on the ESXi console, I managed to get the management IP back up and pingable, and was able to connect to the server with the vSphere client.  I did this by X'ing both vmnic0 and vmnic1 in the interface config, then setting VLAN 300 in the VLAN configuration.  But, I wasn't able to properly team/bundle the vmnic interfaces into both vSwitches.  I could only ever attach one vmnic to one of the switches.

Can I do all of this with only 2 GigE interfaces and retain access to the management IP?

CISCO SWITCH <== 2xGigE w/VLAN 200,300 trunk ==> ESXI SERVER

VLAN 200 is a public /28

VLAN 300 is a private /24 (eg: 192.168.100.0/24)

Management IP is 192.168.100.2

Do I need to create a third VLAN for the management IP and move it?  If I trunk, say, VLAN 400 down to the ESXi server, use another 192.168 block for its IP, will I be able to then take vmnic0 and vmnic1 and team them on BOTH vSwitches?

Still following me? ... 🙂  If I can clarify any of this, by all means ask.  I apologize for the sort-of random babbling here.  Thanks!

jas

[nielse]

You currently have your ports in access mode which limits them to 1 VLAN. You could put them in trunk mode and have more VLAN's available per physical NIC.

Are you trying to bundle the NIC's on a vSwitch? As you can only assign a physical nic to 1 vSwitch you will probably run into a problem here and will have to buy an extra NIC card so you can create 2 NIC's in 1 vSwitch.

[rickardnobel]

jasonvp wrote:

After struggling through the menu-driven UI on the ESXi console, I managed to get the management IP back up and pingable, and was able to connect to the server with the vSphere client.  I did this by X'ing both vmnic0 and vmnic1 in the interface config, then setting VLAN 300 in the VLAN configuration.  But, I wasn't able to properly team/bundle the vmnic interfaces into both vSwitches.  I could only ever attach one vmnic to one of the switches.

It is very possible to do what you like. However, you should use the vSphere Client and not the ESXi DCUI console, it is only for setting initial configuration or repair purposes.

You can not have your two vmnics (physical NIC ports) connected to two vSwitches and at the same time have any "teaming". You will have to remove one of the vSwitches and recreate the portgroups on the remaining vSwitch. The VLANs will still isolate the different networks.

Have you access to the physical switch? As noted above, the ports must be in "trunk" mode, instead of access. And before setting it to trunk you must also reconfigure the portgroups and select the VLAN number.

Before doing anything, could you also attach a screenshot of your network tab in the vSphere Client?

[jasonvp]

Rickard Nobel wrote:

It is very possible to do what you like. However, you should use the vSphere Client and not the ESXi DCUI console, it is only for setting initial configuration or repair purposes.

I was only driving the server from the console to try and set the management IP up to use the same vmnic[0,1] Ethernet bundle on VLAN 300.  Once I had L3 connectivity to that IP, I closed the console and moved to the vSphere client on another machine.  It's there that I got a bit wrapped around the axle, as it were...

You can not have your two vmnics (physical NIC ports) connected to two vSwitches and at the same time have any "teaming". You will have to remove one of the vSwitches and recreate the portgroups on the remaining vSwitch. The VLANs will still isolate the different networks.

Bear with me here as I'm at the office now and not in front of my ESXi server @ home.  But, if I'm grokking what you've written, you're saying that in the vSphere client:

1. Nuke vSwitch 2
2. Team both vmnics on vSwitch 1
3. Assign original ports on vSwitch 1 to VLAN 200
4. Assign new set of ports on vSwitch 1 to VLAN 300
5. Attach VMs to appropriate port groups as needed

Does that sound right?  Will doing that still allow me external access to the ESXi management IP which is also on VLAN 300?

Have you access to the physical switch? As noted above, the ports must be in "trunk" mode, instead of access. And before setting it to trunk you must also reconfigure the portgroups and select the VLAN number.

I do, and I fully understand creating the LAGs and VLAN trunks.  Before attempting to reconfigure the ESXi server, I created a new LAG on the switch, and then set that LAG to mode trunk with both VLANs (200 and 300).  As it turns out, I'm a network engineer by trade. 😉

Before doing anything, could you also attach a screenshot of your network tab in the vSphere Client?

Not at the moment as I stated: I'm at the office and not home where I can do that.  But I appreciate the help!

jas

[Gkeerthy]

if your objective is to achieve link aggregation then the process is below

- create etherchannel (combine 2 pswitch ports) and set trunk mode

- create vlan 200, 300 etc and one for vmotion... mgmt etc...

- install esxi.. select first nic....and set vlan for the mgmt..and give mgmt ip

- after setting the mgmt ip, connect to vsphere client... and change the teaming policy to ip hash and add other nic...

- create other portgroups....with vlan...

[jasonvp]

Rickard Nobel wrote:

You can not have your two vmnics (physical NIC ports) connected to two vSwitches and at the same time have any "teaming". You will have to remove one of the vSwitches and recreate the portgroups on the remaining vSwitch. The VLANs will still isolate the different networks.

Thanks for the pointers; I finally had an opportunity to try this out and it's working as expected.  I'd assign you the "correct answer" but apparently the forum won't let me since you already have a "helpful answer".

jas

[rickardnobel · Accepted solution]

jasonvp wrote:

Rickard Nobel wrote:

You can not have your two vmnics (physical NIC ports) connected to two vSwitches and at the same time have any "teaming". You will have to remove one of the vSwitches and recreate the portgroups on the remaining vSwitch. The VLANs will still isolate the different networks.

Thanks for the pointers; I finally had an opportunity to try this out and it's working as expected.  I'd assign you the "correct answer" but apparently the forum won't let me since you already have a "helpful answer".

Nice that you got it to work! When doing the actual configuration with vSwitch IP Hash and physical switch LAG config it could be a bit difficult to do things in a correct order to not lose connection to the ESXi host.

You could select this message if you like.