VMware Cloud Community
simsync
Enthusiast
Enthusiast

Virtualised ESXi - Network issues

Hi everyone,

I'm having some troubles setting up a lab environment. I know that a virtualised ESXi isn't fully supported but I also know that when I took one of the VMware classed, our labs were all virtualised Smiley Happy

It'll be in Hyper-V, I'm able to boot the ESXi and everything else related, what I think I'm having trouble with is the networking.

So, imagine that I have 3 virtual Switches called EXT01, INT01 and INT02. Now, EXT01 is connected to my own network where the physical server is connected to and INT01 and INT02 are two internal networks.

I have Win 2012 R2 VM which is configured as DHCP/DC/DNS server and is connected to both INT01 and INT02.

For INT01, the DHCP scope will release the following IP/Config:

192.168.100.200-250, 255.255.255.0, NO Gateway and 192.168.100.100 as DNS.

INT02:

192.168.101.200-250, 255.255.255.0, NO Gateway and 192.168.101.100 as DNS.

The Windows box is configured with a static IP on both the NICs (192.168.100.100 for INT01's NIC and 192.168.101.100 for INT02's NIC).

With this said, the Hyper-V host is able to get an IP from both the NICs and any other VM I connect to is able to get an IP and communicate with this server (browsing shares as well).

Finally, the ESXi (5.5.0):

It has 3 NICs, one connected to EXT01, one to INT01 and the other one to INT02. EXT01 is automatically enabled as management whilst the other two are disabled in ESXi. With this configuration (EXT01 is a fully functional network) I get an IP and I'm able to reach the ESXi through vSphere Client or via HTTP and ping.

When I try to set up a new vSwitch for management for INT01 or 02, I don't get any IP and if I manually set it, it doesn't work.

I then tried to disconnect EXT01 from that NIC and I connected INT01, then restarted the management network. I now get an IP from DHCP but guess what? Can't reach it with a ping or via http or via vSphere Client. I can't reach anything from the ESXi either. Same for INT02. When I connect it back to EXT01, I'm able to access the ESXi and the ESXi is able to run a ping test successfully.

Any clue please?

Tags (3)
17 Replies
rcporto
Leadership
Leadership

Try enabling the MacAddressSpoofing on virtual network adapter used by your vSphere ESXi VM: https://technet.microsoft.com/en-us/library/jj679878.aspx#bkmk_mac

---

Richardson Porto
Senior Infrastructure Specialist
LinkedIn: http://linkedin.com/in/richardsonporto
simsync
Enthusiast
Enthusiast

Thanks for your suggestion. I tried, but it didn't work.

The ESXi is still unreachable when connected to either INT01 or INT02. I tried to restart the management network as well as rebooting the machine after the change.

Reply
0 Kudos
cykVM
Expert
Expert

Hi,

I'm a little confused about your configuration and what you expect from it. I would consider this as expected behaviour if you disconnect your management network and connect the interface to a just-internal switch.

Maybe I'm not getting your point in that, so tyr to explain what you want to accomplish. In addition to that: For what reason are you setting up a multihomed DC?

Some basic setup instructions for a multi-homed Windows 2012 server are found here: Network Interface Configuration for Multihomed Windows Server 2012 DirectAccess Servers | Richard Hi...

(especially the part about static routes might be of help with your setup)

cykVM

Reply
0 Kudos
simsync
Enthusiast
Enthusiast

Hi and thanks for your reply.

I've had a look through the link you shared and unfortunately it doesn't tell me anything new, but thanks again.

I will try to be more specific. I made a quick drawing as well on how this has to look.

So, I have a physical box connected to the office network. This physical box is running Hyper-V and has got 3 virtual switches. One External (EXT01, connected to the office network through one of the server's physical NICs), the other two are Internal switches (not private, so they, once created, will create a virtual NIC on the Physical box automatically; in my case I will have 2 virtual NICs added to the physical host) called INT01 and INT02.

In Hyper-V I have a W2K12 R2 VM running the following roles:

- Domain Controller

- DNS

- File Services

- DHCP

I have connected two virtual NICs to this VM:

- One connected to INT01

- One connected to INT02

As there are no network devices on both internal networks, I manually added the following config for INT01:

IP 192.168.100.100

Subnet 255.255.255.0

Gtw: left blank

DNS: 192.168.100.100

INT02 on this VM has got the following config:

IP 192.168.101.100

Subnet 255.255.255.0

Gtw: left blank

DNS: 192.168.101.100

So far so good.

Now, I created two scopes, one serves 192.168.100 addresses and the other one takes care of 192.168.101.

The scopes will both provide the same subnet (255.255.255.0), they'll leave the default gtw blank and they will also provide a DNS server address which match the VM's NIC IP, so 192.168.100.100 for INT01 and 192.168.101.100 for INT02.

As soon as I configured the DHCP scopes and ran ipconfig /release and /renew on the Hyper-V host, its two virtual NICs I talked about above, got an IP address (192.168.100.200 for the vNIC connected to INT01 and 192.168.101.200 for INT02).

I can access the VM through 192.168.100.100 and 101.100 with no problem, I can ping it, RDP on it, access its shares, all from the physical Hyper-V host.

So far, everything looks as it should and works as expected.

Now, one I set up a new ESXi and connect it to either INT01 or INT02, it gets an IP address from the DHCP, but then it dies there, not reachable what so ever whilst my other servers can talk to each other.

The reason why I want the ESXi to be connected to 3 networks is the following:

- EXT01 to be used as management and will be accessible from my office

- INT01 to be used for vmotion etc

- INT02 to be used for the ESXi's VMs (I want any VM created on the ESXi to get an IP from the external 2k12 box)

I don't need to access other subnets from the internal networks, so I'm happy to leave the traffic as local. So when I'm pinging 192.168.100.100, traffic will only go through INT01, when I ping 10.0.2.50 (my office network), traffic will go through the EXT vSwitch and when I ping 192.168.101.100, traffic will have to go through INT02. And this works on 2k12 boxes..

Here the PIC

Untitled.png

Reply
0 Kudos
cykVM
Expert
Expert

If you did not miss anything in your drawing and config details, I would assume that there is simply no route to the management network of the nested ESXi. Since both physical NICs (NIC1 and NIC2) are connected to the office network (I guess both have a 10.0.2.x IP?) it tries to route the ESXi management traffic through NIC1 which has no connection to the EXT01 management vSwitch. Therefor ESXi management is not reachable.

Reply
0 Kudos
simsync
Enthusiast
Enthusiast

This seems weird though, why does it work in Windows?

Also, NIC1 and 2 are both connected to the office network, correct, but only NIC2 is the one connected to the EXT01 vSwitch and if you see above, I've tried to disable all NICs on the ESXi host and enable just 1 for INT01 and yet I wasn't able to reach it. It would get an IP but nothing more.

Anything I can do in this configuration that allows me to have the ESXi reachable from 3 different networks?

Ultimately, if you think about it and remove the Physical Host and consider NIC 1 and NIC 2 as a standard port on a physical switch, I would still face the problem of trying to have an internal network with no gateway.

Reply
0 Kudos
bharathl
Enthusiast
Enthusiast

I think you are not simply connecting to a virtual adapter when you are connecting to ESXi server but you are connecting the internal virtual switch in Hyper-V to another Virtual switch in ESXi server. As per the following definition of Internal switch I think you can not use the internal switches for this purpose.

Internal Switch

The internal switch is similar to the private switch with one exception: the management operating system can have a virtual adapter on this type of switch and communicate with any virtual machines that also have virtual adapters on the switch. This switch also does not have any matching to a physical adapter and therefore also cannot uplink to another switch.

Reply
0 Kudos
cykVM
Expert
Expert

How many vSwitches are configured in your ESXi host?

I think it's all a matter of a clean networking config inside ESXi. As you have separated the Management Network and the VM Network (and the future vMotion Network) there is just no route between them. If you now disconnect/disable all but the VM Network uplink the Management Network does not have any IP anymore.

Reply
0 Kudos
simsync
Enthusiast
Enthusiast

It does. If I disconnect all of the NICs, and then connect on NIC1 INT01, it does get an IP address from the 192.168.100 range. It also get the right subnet mask and no gtw.

This clearly shows that it is connected.

I've also tried to build an ESXi with just one NIC connected to just INT01, same story, I do get an IP but when I try to reach the host (from the DHCP server that released that IP!) I don't get any reply.

Cheers

Reply
0 Kudos
cykVM
Expert
Expert

Have you tried a traceroute (or tracert in Windows) to the ESXi IP provided by DHCP? Maybe that shed some light into where the connection stops/is blocked.

Reply
0 Kudos
cykVM
Expert
Expert

Another thing come to my mind: Did you enable promiscious mode for the nested ESXi at least on the VM network?

Some further reading here: VMware vSphere nested in Hyper-V on Windows 8 • My Virtual Vision

And this discussion: Vmware ESXI within Hyper-v (server 2012) networking issue

simsync
Enthusiast
Enthusiast

Thanks for the links. I've seen them both already but only tried to add the config to the xml file now, with no luck.

However, and this is weird, I've created a second ESXi host, whilst the first was still up and connected only to INT01. I connected the ESXi02 to INT01 as well, and guess what?

I can ping the two ESXi within each other but can't ping their DHCP server lol

I've also tried to ping the DHCP server using the other NIC (101.100) just to exclude any issue.

Capture.jpg

And I've also checked the bindings which are correct (in DHCP):

Capture.jpg

Traceroute from Windows:

Capture.jpg

Reply
0 Kudos
simsync
Enthusiast
Enthusiast

Guys...

I think I moved a bit forward. As you know, in order to have the nested ESXi to recognize the virtual NICs, you need a Legacy adapter I didn't think of adding a Legacy adapter to the virtual NIC of my 2k12 VM.

It worked as soon as I removed the std NICs from the 2k12 VM and added the legacy adapters. I was able to ping both the ESXi hosts. Now, I need to understand if the change I made in the XML has to be done as well or not.

This is from the DHCP Windows VM:

Capture.jpg

The only problem I have but I don't care of, is that I can't change (I think) the type of vNIC on the Host itself, so the host can't see it. To be honest I don't really care, what I can do is, I can make them private switches and remove the vNICs from the Host as I don't really need to access them through the host.

simsync
Enthusiast
Enthusiast

Right, this is endless. It only works when the first vNIC is connected to any of the networks. When I try to create a new Management Network in vSphere it just doesn't an IP or anything. I think it's being disturbed by the active GTW of my EXT network?

Reply
0 Kudos
cykVM
Expert
Expert

Maybe an update to yout diagram/drawing would help us to follow you.

I still think, that the 2 physical NICs connected to your office network will cause problems. Additionally I would first simplify the whole design to only 1 INTernal network and after this is working annd the 2nd one.

Reply
0 Kudos
simsync
Enthusiast
Enthusiast

I finally made it!

The second issue I was having was a distraction: when I rebuilt the ESXi from scratch (VM itself), I forgot to enable the MAC Spoofing :smileyshocked:

I've been showing it on the phone for like 30 minutes to a colleague of mine, you know, a couple of eyes more and when he asked to see the VM configuration I realized myself what was missing. Basically when I firstly had this set up, it didn't work because I didn't have mac spoofing, promiscuous mode and a legacy adapter on the 2k12 machine. Then I had all done as you guys suggested but then by mistake I forgot about the spoofing..

So, to make it short I:

  • Removed the two standard virtual NICs from the DHCP/DC/DNS W2K12 VM and add Legacy NICs. [No need to enable MAC Address Spoofing on these two network cards]
  • Created the ESXi in Hyper-V and make sure each NIC was a Legacy NIC. Furthermore, I enabled MAC spoofing in the advanced features tab.
    • Connected the first one to EXT01
    • Connected the second one to INT01
    • Third on to INT02
  • Removed, from the ESXi in Hyper-V, all Integration Services (kind of useless to keep them on, I only left the volume checkpoint one on)
  • Enabled promiscuous mode in the ESXi's VM's XML, right under <global_settings>
  • Installed the ESXi on the newly created VM
  • Connected to the ESXi through the EXT01 address (which is the only one with a gateway, from my office)
  • Configured the networking as I like
    • So I removed from the standard vSwitch0 the VM Network port group and also disabled Management Traffic as I want this to purely communicate with my office network only to allow me access to the ESXi.
      • I might come in the future (if I want a vCenter VM running in VMware) and add a VM Network again for External access so that I can also access vCenter from the office network
    • I also created two new vSwitches, one for INT01 which I will use for vMotion, FT etc and one for INT02 for VM traffic

I hope I won't encounter any other issue during my testing, one thing I can tell for sure: Having this lab nested is great as I can easily restore a previous configuration in case I break something during my testing.

Now head down to study!!! Smiley Happy

Thanks everybody for helping!

Reply
0 Kudos
cykVM
Expert
Expert

congratulations that you made it work:smileycool: and thanks for the update on your so far working configuration.

cykVM

Reply
0 Kudos