Host Disconnect after migration to vDS

[architects]

Hi,

I created a vDS called Management & vmotion and connected it to 2 1 Gig uplinks. I then created 2 port groups, one for SC and one for Vmotion. The VLAN ID and ip address for both are correct. I was able to ping and add these new hosts to the cluster when they were at vswitch before however as i moved them over to vDS the hosts got disconnected one after the other.

Now, all the hosts are disconnected from the VC however i can login to each of them individually and ping all of them from one another. I cannot ping the gateway or any other IP fropm these hosts now. Seems like they fell off the network after adding to vDS.

Has anyone seen this before. Is there an update on the host that i can manually modify and have them see the gateway again?

Any ideas would be appreciated.

Thanks,

A

[Linjo]

Is the vCenter also a virtual machine on the same host?

Best regards,

Linjo

If you find this information useful, please award points for "correct" or "helpful".

[architects]

Hi Linjo,

No. The VCenter is a seperate physical box.

I added this 3 host vsphere cluster today & it was working fine till the time my SC was sitting on vswitch vs. vDS. I created a new vDS for SC and mapped it to uplinks. Then used the migrate wizard to migrate the SC port groups from vswitch into this. Sometime after this all the hosts fell off the network. Now, i cant reach the Default gateway ip.

This is on a IBM blade center. The hosts are using onboard bComm nics to connect through a cisco 3012 switch on the blade center and finally plugging into the Nexus 5020'a we have in the DC.

Let me know if you need more info.

Thanks,

A

[chadwickking]

Hi,

If you run esxcfg-vswif -l does it list your serice console and gateways?

(Ran directly on the host)

Also there are some best practices in doing this in the best ways: take a read on this blog:

http://mike.thegrahams.ws/?p=3

Did the Pnics you added to the DVSwitch have the current configurations and Vlans as well?

Cheers,

Chad King

VCP-410 | Server+

Twitter: http://twitter.com/cwjking

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[chadwickking]

Are these EX server local or over the WAN? We ran into an issue with running vDS over the wan with ESX servers.

Cheers,

Chad King

VCP-410 | Server+

Twitter: http://twitter.com/cwjking

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[peterdabr]

It almost made me think as if the uplink group on dvswitch had no SC vlan # added to it, but then you stated hosts can ping each other's SC IP which means both, dvswitch uplink and ports on the physical switch that all hosts connect to do have proper vlans added to the trunks.

I think you need to start from the scratch by manually creating another 'test' dvSwitch with only a single uplink and only one port group being the SC. Then choose one of the esx hosts and connect it to the new dvswitch and try pinging/arping (arp -an from SC of that host to see if you can obtain MAC addresses) the default gateway and other physical servers on the same LAN, prefferably connected to the same physcail switch as the esx hosts.

Peter D

[architects]

Hi,

The ESX servers are local and not over the WAN. I did ran the esxcfg-vswif -l and here's the acreenshot attached. I dont see the gateway listed however it was configured.

Not sure..

A

[architects]

Hi,

Yes, i was thinking about creating from scratch however the problem is that since all the hosts are disconnected from the VC i cannot create a vDS since there would be no hosts to add and hence no uplinks.

Is there a procedure to do this via comamnd line?

Regards,

A

[peterdabr]

You can disconnect esx host from dvSwitch by running this command:

esxcfg-vswitch -Q vmnic# -V <dvPort> <dvSwitch>

This command will unlink vmnic of an esx host from dvswitch.

Peter D

Update: I hit sent to soon

Once you have freed up esx host from dvswitch create a new vswitch using and add sc port to it. Then connect it back to vcenter for further testing with newly created test dvswitch.

Here is the list of all commands on 'broken' esx node:

esxcfg-vswitch -Q vmnic# -V <dvPort> <dvSwitch>

esxcfg-vswitch -a vSwitch0

esxcfg-vswitch -A "Service Console" vSwitch0

esxcfg-vswif -a vswif0 -p "Service Console" -i <ip addr> -n <mask>

esxcfg-vswitch vSwitch0 -p "Service Console" -v <vlan number>

esxcfg-vswitch -L vmnic0 vSwitch0

[architects]

Hi Peter,

An interesting observation. I tried to list the vswitch membership of the ESX and it showsd my VDS (vDS Managemnet) where my SC port belongs however when i go in to delete it says no such DVS found. Attached is the screenshot for you until i try some more options. By the way, i tried, different alterations like i used different vmnic's, changed the vmnic# to vswif# and so on but same problem. It doesnt like my vDS.

Any ideas?

[peterdabr]

If you take a look at the screenshot you'll notice your vswif0 (SC) is on dvport 102. That's what you have to use as -V parameter

Peter D.

[architects]

Tried all the permutations earlier...no luck...take a look. Also attaching the VC side of configuration for a second set of eyes...

Regards,

A

[peterdabr]

Your dvswitch has a space, so use quotation marks (single or double...try both as I'm not sure which one is correct), example:

-Q vswif0 -V 102 "vDS

Management"

Also, in order to completely free up esx host from the switch, make sure to do the same for all dvports that are active now

Peter D.

[peterdabr]

vcenter screenshoot looks in order...just make sure 'Management Uplinks' has a vlan range that both vlan34 and 500 falls in. Other than that, start troubleshooting by doing everything from the with new test dvswitch with one uplink, one esx host connected to it and a physical host on the same vlan/netblock, connected to the same physical switch.

[architects]

Hi Peter,

deleted everything on the VDS side, created a new vswitch and attached a vswif to it with the right ip, netmark, gateway etc. and connected only 1 uplink.

Still unable to ping it's ip from my laptop, cannot add it to the VC.

Also, i now cannot even ping this from other 2 ESX hosts. I tried to ping the gateway from this host and that also failed. I am now beginning to worry that this might be a network problem in the DC vs ESX issue.

Do you think so as well or can we try something else?

Regards,

A

[chadwickking]

Sorry I was away from earlier 😕

can you verify you can see gateways on the vswifs again? - thanks trying to draw a good picture - this happens from time to time with migrations and using Dvswitches if its not carefully thought out. Can u also confirm that trunking is set correctly on the Physical switch as well?

Cheers,

Chad King

VCP-410 | Server+

Twitter:

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[peterdabr]

It has to be a network problem. With a laptop connected to the same switch as the hosts can you ping def gw?

Also, just a tip, always try running arp -an in addition to ping to see if you can obtain mac addr of the gw as sometimes imp traffic can be blocked and needs explicit ACL if for instance it is an interface on a cisco device. With arp you should always see Mac address of dest device, if both can 'see' each other

Sent from my iPhone

[architects]

Hi Pete,

I did try the arp -a and got the mac of the router. So i am fine. Also, tried to ping it from different sources out of the ESX and it pings fine. I think i gotta get the network team involved or reload one of my ESX hosts to start eliminating things one at a time.

Chad: I do see the swif with gateway address attached & am i assuming the trunking is correct since this was working fine before i added the vDS and moved them from SC Vswitch into vDS mnagement switch.

Running out of idea now....would reload ESX and talk to network folks. Let me know if u got more ideas else i would contact you on Monday and give you an update of what we did

But, thanks for all the great advice!!

Regards,

A

[chadwickking]

You still having an issue or did you get it resolved over the weekend? Keep us posted as well on your progress!~

Cheers,

Chad King

VCP-410 | Server+

Twitter: http://twitter.com/cwjking

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[architects]

Hi Chad & Peter,

Sorry for being out of touch on this one. I was neck deep into something else and this too. We finally figured out that it was indeed a network problem and nothing to do with ESX.

There was one VLAN that wasn't accessible due to which anything on it was going crazy. I did try to disable VLAN and it worked sometimes anf failed some. So i finally ended up flashing the blade n/w module firmware, nic drivers & reloaded OS with no VLAN on one host and everything seems to be fine now.

Thanks for all your good suggestions!

Regards,

A