I think I'm losing my mind. Here is my scenario: I originally had three ESXi 3.5 U4 hosts in a cluster managed by virtualcenter 2.5. A few weeks ago I did a successful upgrade on VC to version 4.0. All of the configs stayed the same. All of my network adapter/virtual switches were configured the same on all hosts:
vSwitch0 / VMkernel Port / Management Network / vmk0: 192.168.1.x (16 - vmsvr1, 17 - vmsvr2, 18 - vmsvr3)
vSwitch1 / Virtual Machine Port Group / Virtual Machine Network
vSwitch2 / VMkernel Port / iSCSI VLAN 25 / vmk1: 192.168.25.x (13 - vmsvr1, 14 - vmsvr2, 15 - vmsvr3)
vSwitch3 / VMkernel Port / iSCSI VLAN 26 / vmk2: 192.168.26.x (13 - vmsvr1, 14 - vmsvr2, 15 - vmsvr3)
Here is my IP scheme for the physical ports on the servers: Nic1: 192.168.0.x - virtual machines. Nic2: 192.168.1x - Management network, Nic3: 192.168.25.x - iSCSI VLAN, Nic4: 192.168.26.x - iSCSI VLAN. Both iSCSI VLANS have access control lists that do not allow traffic to leave or enter the VLAN. In other words you cannot access VLAN 25 or 26 from any other network like 192.168.0.0 or 192.168.1.0.
Yesterday I evacuated all of my VM's from my first ESXi host: vmsvr1. (my other hosts are vmsvr2 and vmsvr3) I put the host into maintenance mode and moved the host outside of the cluster. I then deleted this host from VC. I did a fresh install of ESXi (embeded) onto my Dell PowerEdge R805 server. After configuring the management interface exactly like the original server I configured the network settings on vmsvr1 exactly the same as above. I did all of the other configs the same as well (scratchconfig, iSCSI SAN, etc). I added the host vmsvr1 to the cluster on my VC. Then got an error when configuring the host for VMware HA:
Host vmsvr1 does not have the following networks used by other hosts from HA communication: 192.168.25.14,192.168.26.14. Consider using HA advanced option das.allowNetwork to control network usage.(FYI: those addresses are my iSCSI adapter addresses for vmsvr2 - are not NOT configured for HA comminication)
Host vmsvr1 has the follow extra networs not used by other hosts for HA comminication: 192.168.1.16. Consider using HA advanced option das.allowNetwork to control network usage. (this IP address is my management network along with 192.168.1.17 for vmsvr2 and 192.168.1.18 for vmsvr3)
Now the REAL kicker is this: When I go and look at my network configs for vmsvr2 I noticed that it changed from the config I posted above. Vmsvr2 now looks like this:
vSwitch0 / Virtual Machine Port Group / Management Network
vSwitch1 / Virtual Machine Port Group / Virtual Machine Network
vSwitch2 / VMkernel Port / iSCSI VLAN 25 / vmk1: 192.168.25.14
vSwitch3 / VMkernel Port / iSCSI VLAN 26 / vmk2: 192.168.26.14
What the heck is going on here? I did not change anything on vmsvr2. Even thought it shows vSwitch0 as a VM port group with no IP address I'm still able to manage the machine as if it were a VMkernel port with an IP address assigned to it. This is why my HA config is bombing on vmsvr1. What is causing vmsvr2 to change on it's own and how can I change vSwitch0 back to a VMkernel port with an IP address without messing everything up?
Thanks in advance,
Nick
