ESXi 5.5 Network VLANS with limited NICs setup ?

[stanj]

I am working with  Simplivity Omicube which is an all-in-one solution.

Typically, I run mgt on one VLAN, SAN (Data Access) on another, VMotion, etc.

However, this box has only two 1 Gig Ports.

A new network scheme was given to us that needs to be configured as

193.168.51.0/24 - VLAN 210          

GW 193.168.51.1         

ESXI is on vSwitch 0 at 193.168.51.10

GW 193.168.51.1

193.168.52.0/24 - VLAN 220          

vCenter is targeted for 193.168.52.10          

View Connection Server 193.168.52.20           

GW 193.168.52.1

193.168.53.0/24 - VLAN 230           

GW 193.168.53.1           

Domain Controller is targeted for 193.168.53.10

Given the above, what steps are required to set up the virtual network to have the VMs communicate with each other across

VLANS given that they are on different subnets and only one more physical NIC available?

Would this be set up via nic teaming

thanks

[vfk]

You can have all three vlans trunked on both links, created a portgroup for each vlan and set the vlan on the portgroup.  Communication between the vlans will be handled a router or the switch itself if it is capable of routing.  Use both nics as uplinks and the default teaming policy Route based on the originating port ID is sufficient for this configuration.

How are presenting storage?  Over the same two nics...iscsi / nfs??

[stanj]

thanks I am not sure how the storage is presented in the OmniCube.  It is a new product we are working with and runs an appliance to manage the storage.

We have another engineer who has worked with the OmniCube and will be looking at that side of the configuration. He is out of town at ht present time.,

as to the configuration, the network admin is using a Cisco switch and I believe he said that there was, at one time, an 802.1Q trunk set along with the VLANS.

However, when I  tried to ping or log into the ESXi Server, I got no response.

So he made the connection to the ESXi server a single VLAN indicating this was needed because the ESXi was not configured to accept a trunk?

He indicated, that if he put a trunk on the switch, there would be no connectivity via the physical port until I configure the internal switch for trucking.

are you saying to set the portgroup on vswitch0 for ESXi to VLAN 210 and set the NIC Teaming to Route based on the originating port ID , and also add a second nic along with the other portgroups / vlans?

thanks

[vfk]

Yes, if both uplinks are 802.1Q trunk, you just need to tag the portgroups and vmkernel interface with the appropriate vlan.  The alternative is to ask the network guy to set the management network vlan as native vlan on the trunk.

For Example:

[stanj]

Ok,

the network admin currently does not have the 802.1Q trunk,set,,

he has the esxi on a separate VLAN.

If I set the portgroup on vswitch0 for ESXi (management) to VLAN 210, won't I lose connection to the ESX Server  from the vSphere client?

or

do I create the other portgroups and assign the VLANS and leave the management without a VLAN and tell the network admin to set the network vlan as native vlan on the trunk

I think I recall the the network admin saying it is a chicken and egg thing....

thanks

[stanj]

below is what i have so far,,,the admin has not made any changes at this point..

i did not add or modify the mgt network to VLAN 210 yet,,,I am  thinking the network admin needs to set the 802.1q trunk on the switch?

[vfk]

That is correct, it is easier to ask for the management vlan to be your native vlan on the trunk, and then all your other vlan will be tagged as per your picture.

[a_p_]

>>> ... it is easier to ask for the management vlan to be your native vlan on the trunk ...

That may be an easy task from a configuration perspective, but I would not recommend to do this. Although this may not cause any technical issues, you need to keep this change in mind/document it. I ran into such a configuration with one of my customers and it took a while until the networking team remembered that the native VLAN was changed at some time in the past. However, nobody was able to tell me the reason why.

I'd talk with the networking team and change the Management VLAN-ID right before they apply the change on the physical port. If there's an issue you can always reconfigure the Management Network from the console/DCUI.

André

[stanj]

So, I set the Management PortGroup VLAN to VLAN 210 and did not lose connection to the ESXi Server.

I am not sure what the Admin set on the switch,,he set a VLAN so I could connect the vSphere client to ESXi.

I would now assume if I tell him to set the 802.1q trunk I should be ok,,,

[a_p_]

You will loose connection to the host, at least temporarily until the network admin reconfigured the port to 802.1Q.

The most straight forward and secure way with the two vmnics as shown in your screenshot might be:

• network admin shuts down one of the physical uplink ports and configures it for 802.1Q
• you migrate the VMs to another host
• you ensure that both vmnics are "Active" in the vSwitch as well as in the port group settings
• you set the VLAN-ID on the Management Network (at this point you will be disconnected)
• network admin shuts down the second port and enables the already reconfigured switch port
• you should be able to connect to the host again, if not verify that the Management's VLAN ID is configured properly (from the DCUI)
• network admin reconfigures the second port and enables it again
• you set the VLAN-ID for the VM port groups, which are not configured yet

André

[stanj]

Ok,

So I set the VM Network port group in the above screen to the VLAN 210 and did not lose connection to the ESXi Server.

Not sure why,,

The VMs did lose connection to the internet and each other which seems to make sense since the VLAN is set.

I do not have another ESXi Server to Migrate VMs

[a_p_]

>>> So I set the VM Network port group in the above screen to the VLAN 210 and did not lose connection to the ESXi Server.

You are connected to the host through the Management Network. Changing the VM network has no impact on the Management Network.

You can only successfully add the VLAN-ID to the VM network after the physical port configuration has been done.

André

[stanj]

When you said,,, ......

You can only successfully add the VLAN-ID to the VM network after the physical port configuration has been done.

Where is VLAN-ID added?

In the diagram, I thought this was added in the portgroup setting as I showed ?

I think you are saying, and correct me if I am wrong, even though I set a VLAN ID for each portgroup defined, the VLANS will not communicate with each other until the physical switch is configure 802.1q

thanks

[a_p_]

Yes, the physical switch needs to be configured for 802.1Q first. This way it will forward network traffic without removing the VLAN tag from the packets. Only with the tag in the packet, the vSwitch will be able to forward it to the appropriately tagged port group, which is responsible for removing the VLAN tag from incoming traffic and adding it to outgoing traffic.

André

[stanj]

ok

, thanks,,i will give it a go!

[stanj]

ok,

so I am back from VMworld.

I made changes to the drawing I posted in the previous message.

I removed the vNic since we will be using this to uplink to a new vSwitch connected to a DMZ.

I then set the VM network adapters to the appropriate VLANS.

I asked the network admin then to set the trunk on the CISCO switch that connects the ESXi Server.

I came back later an logged into the laptop I have been using.

I tried to ping the the ESXi Server and get back                   

          Destination Host Unreachable.

I pinged the gateway and do get a response.

I checked vekernel log file and there is a 794 error .. failed to set default gateway ?

I checked the physical nic on the back of the server and it the left light is green and right light off/amber meaning limited to no connectivity.

The network admin said he tried several different settings on the Cisco trunk without success (changing the native vlan and what vlans were allowed on the trunk).

The only way he could ping the ESXi Server was to set the Cisco port up as an access port (vlan 210) versus a trunk port.

So, I am not sure what is going on or if there is some Cisco setting that is being missed?

[a_p_]

When switching from Access ports to Trunk ports on the physical Cisco switch, you need to configure the VLAN-ID on each port group unless the port group is used for the Native VLAN (in this case the VLAN-ID has to be zero/none).

Here's a sample port configuration for the physical uplink port(s) (taken from http://kb.vmware.com/kb/1004074😞

interface GigabitEthernet1/2
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-100
switchport mode trunk
switchport nonegotiate
no ip address
spanning-tree portfast trunk

Some of these settings may not show up on the switch, depending on the model and its defaults.

Once the physical ports are changed, you may loose connectivity to the Management Network on the host, and need to set the VLAN-ID for it on the host's console (DCUI).

André

[stanj]

I will check  again with the network Admin to see if he followed the above steps..

Currently., he set the Cisco port up as an access port (vlan 210)

So, it looks like I will have access via the client

I can have him reset the 210 back to a trunk which means  i will lose connection to ESXi from the client but then

logon the console via tech support mode and run the esxcfg cmd

as in.. (based on the current drawing from above  -i since have set 210 as the vlan on vm network)

esxcfg-vswitch -p "VM Network" -v 210 vSwitch0

thanks

[a_p_]

Why would you want to use the command line? After setting the proper VLAN for the Management Network from the DCUI, you should be able to reconnect to the host, and do the remaining configuration for the other port groups from the GUI.

André

[stanj]

I am not able to access the set up at this point as i am off site.

I am pretty sure i set VLAN 210 for the Management network in the networking section on the vSphere client and when he set the trunk on the switch I lost access to the ESXi Server.

I then went into the ESXi DCUI  (Configure Management Network section) and set the VLAN option to 210 and that didn't work.

I also tried 4095 (all) and still could not connect to the ESXi Server from the client on the laptop or ping the ESXi server.

The other port groups are set with VLANS on vswitch0.

Since i should now have access to the vSphere client because the admin set the Cisco port up as an access port, I will check again..

I suppose much easier than using the console cmd line,,

Ii is possible I made a typo,,but if not, I am not sure where to look next.