Trying to get VLANs working with multiple NICs combined?

[Joe_Papa]

I have ESXI 6.0 installed on an HP ProLiant ML350 G6 server. I'm running pfSense in a VM that is using a 4-port NIC. One port is the WAN and I have combined the other 3 physical NICS together into a Virtual Switch and then created a Port Group with the 3 NICS using VLAN ID 4095 to make it a trunk... I am using the physical NIC as my LAN port and adding two VLANs to the same NIC for IoT devices and for a Guest Network. It isn't working though. I think the VLAN tags are being stripped from my traffic.

** Update ** It seems that having multiple uplinks combined is the problem. Once I reduced the Virtual Switch to one physical NIC (removing the other two uplinks) it worked perfectly. If anyone can explain why this is the case I would appreciate it. Seems kind of arcane/random. 

[nachogonzalez]

Hello @Joe_Papa hope you are doing fine:
Let me ask you a few questions:

Can you share screenshots of your vSwitch configuration please?

Are you using a Standard Switch or a Distributed Switch?

Do you have all 4 Phyisical nics connected and with all VLANs propagated to them?
is there any kind of LAG/port channel configured?
What kind of teaming/load balancing alogrithm is the vSwitch using?

[Joe_Papa]

Here's the configuration that didn't work. 

Here's the configuration of the vSwitch now that it works. This is just the one physical NIC with VLANID 4095. Previously I had the same configuration but with three physical NICS combined as uplinks to this same vSwitch. It would not allow the VLAN tags to function properly.

Here is the configuration of the LAN Port Group

I don't know the answers to all of your questions (I'm only a hobbyist), but I'll try here.

I have Unifi managed switches (not sure if that is standard or distributed). 

The 3 physical NICS were not all in use. I just had them all grouped so it didn't matter which port I plugged my ethernet cable into and so if one port failed I could just move to one of the other two.

I don't have any LAG/port channel configured (not sure what that is). I am just using 2 VLAN IDs on the network and configuring switch ports to control which VLANS can go where on my physical network and over my WiFi

I don't have any teaming/load balancing algorithm activated so it's whatever is default. Sorry I don't have better answers, but thanks for your help and experience. 

 

[nachogonzalez]

Hey, hope you are doing fine

thanks for your replies, I have a better understanding of your situation now.
Let me try to assist you: 

I have Unifi managed switches (not sure if that is standard or distributed).  --> vSphere networking's switches can be both standard and distributed (check this) you are using a standard switch.

 The 3 physical NICS were not all in use. I just had them all grouped so it didn't matter which port I plugged my ethernet cable into and so if one port failed I could just move to one of the other two. --> I assume you are using a homelab, this is not the ideal situation. 
ESXi  is capable of "moving the cable" by itself without human interaction, but for that purpose you will need all 3 uplinks actively connected at the same time.

I don't have any LAG/port channel configured (not sure what that is). --> In a few words It's a way of making two nics work as one for improved throughput and resiliency/fault tolerance.
https://en.wikipedia.org/wiki/Link_aggregation

I am just using 2 VLAN IDs on the network and configuring switch ports to control which VLANS can go where on my physical network and over my WiFi --> I'm not sure if I understood this: you have different VLANs assigned to different NICs? That would not be ideal neither, you can do it with the vSwtich portgroups

I don't have any teaming/load balancing algorithm activated so it's whatever is default. --> Your policy  is route based on Originating virtual port id.

Let's do this:
Add the 3 adapters but then select only one as active and the others as standby.

If you need further assistance let me know.

[Lalegre]

Hey @Joe_Papa,

Your case is a little bit hard to analyze as you are mentioning that you pfSense is a Virtual Machine with 4 vNICs but then you are sharing some images for the pNICs that are connected to the ESXi which is not the same as vNIC.

I understood that your pfSense has one Uplink connected to the WAN portgroup which I suppose is for providing connectivity to outside the virtual infrastructure but then I got lost on the LAN configuration.

Always remember that if you are creating portgroups in vSphere in Trunk mode then you should tag the VLANs inside the pfSense for the traffic to be able to flow.

Could be possible to do a quick diagram of what do you want to achieve so we can have more insight on your configuration?

[Joe_Papa]

Ok. I'll try to be more precise. pfSense is set up in a VM on this server. I have a 4-port GB NIC available which I have allocated to pfSense as vSwitches (not passed through). I have set up pfSense with the following interfaces; WAN, LAN, VLAN(30), and VLAN(50). LAN is for local secure traffic, VLAN(30) is for IoT devices, and VLAN(50) is for a Guest WiFi.

So technically, I only need 2 NICs allocated to this VM as all 3 internal networks can function over one NIC. See below.

However, since I have 4 ports available on the physical NIC, and I don't need them for anything else, and it's easier for me to keep them straight if they are all dedicated to this task... I created one LAN vSwitch in ESXI with 3 physical NICs all added as uplink. I thought this would make all three NICs effectively interchangeable so my routing would function properly as long I had my physical managed switch connected to one of the three NICs that were combined for this vSwitch. And it did work like this before I added the VLANs. Once I added the VLANs and configured my managed switches properly for the VLANs I was not able to route any traffic that was tagged. I assumed that this had something to do with the way ESXI behaves when combining the three NICs in one vSwitch... But after further trial and error that doesn't seem to be the case. 

Update- I removed 2 of the NIC uplinks from the vSwitch and entered VLAN ID 4095. Restarted the pfSense DHCP service. And voila it worked properly with both VLANs being properly recognized and all segregation rules functioning. So I assumed that the multiple uplinks was the problem. After letting it run for a day or so though I tried combining the three NICS again and it is up and still working fine. Now I think that it was just the VLAN ID that was the issue.