VMware Communities : All Content - VMware vSphere™ vNetwork

Can't Remove Host from dvSwitch

xavier@lynk.com — Wed, 25 Nov 2009 17:38:20 GMT

I was getting the same message "The resource 119 is still in use. DVS dvSwitch1 port "119" is still in use on host esx-host port <not-found>" when trying to remove my host from the DVS switch so I went around it by right click on the ESX host and disconnect then I was able to remove the host from the DVS switch

If this was helpful please assign points--

Vsphere Service Console VLAN; Command Line ??

mike.laspina — Wed, 25 Nov 2009 16:18:04 GMT

Yes, confirmed.

http://blog.laspina.ca/
vExpert 2009

VM's can't ping network but can ping host

Bootrix2 — Wed, 25 Nov 2009 13:19:33 GMT

More information:

I created vSwitches for each VM with two nics each per switch. (The Amount of Nics is irrelevant).

It appears that the primary machine to access the two ports would have priority and any other subsequent VM on the same vSwitch couldn't get out.

So, VMswitch1 with VM_1 and VM_2. If VM_2 became active before VM_1 it would seize the nics. VM_1 now online can ping itself and ping VM_2 but can't ping the host/service console (its on it's own vSwitch).

So essentially, I thought it was vSphere but it points more towards my Cisco switch that's not allowing more than 1 mac address to route traffic through shared ports.

I've yet to bother changing the switch settings.

Can't ping VM, but Kernel on it's vSwitch

JoeAKim — Wed, 25 Nov 2009 08:33:03 GMT

We've tried a new NlC, that didn't do it. (HP NC360T)

So, here's the whole problem once more::
Server: HP DL380 G5
OS: ESXi 4.0
1 On-Board NIC, Dual-Port.
1 PCIe NIC, Dual-Port.

The 1st port of the On-Board Port works fine. Every other time I restart the server, the 2nd On-Board Port has the same problem as any PCIe-Card.

When I create a new vSwitch, connecting to the 1st port (Mainboard), everything is fine. When I create a second vSwitch, connected to either the 2nd (Mainboard)
port, 3rd or 4th (extra-Card) port, and connect one of the VMs to it, the VM can't neither see the network, nor can it be pinged from outside. Now, if I add a VMkernel to that vSwitch, the Kernel can be pinged without problems, but the VM is still isolated.

I looked at the lights of the NIC, it flashes green at the beginning, later on the orange LED glows permanently. And I know that IP-Adress, Subnet-Mask, Gateway of the VM and the server are ok. Otherwise I wouldn't be able to ping (etc.) the VM when it's connected to the 1st NIC.

I already called HP, but they say when I can ping the card, then it's a VMware problem.

I'm sorry, I can't provide any screenshot, since I already took the PCIe Cards out of the server.

Thank you.

Network Failover Issue

dlietz — Tue, 24 Nov 2009 19:03:12 GMT

Switching all of the vswitches to failover using 'beacon probing' seems to have done the trick. Thanks for the help everyone.

Virtual Switching Vs. HP Blade Switches

jpdicicco — Tue, 24 Nov 2009 15:51:42 GMT

Best practices are to have a dedicated NIC for your Service Console and a dedicated NIC for your VMKernel interface (mainly for vMotion).

A 4 NIC host usually looks like this:
vSwitch0
The SC portgroup using vmnic0 active and vmnic1 standby
The VMK portgroup using vmnic1 active and vmnic0 standby
(NB: In some hardware, it is better to use vmnic2 than vmnic1 for redundancy. Evaluate your own system architecture for single points of failure.)

vSwitch1
VM portgroup(s) with both NIC active or active/standby based on your needs and policies

JP

Intel PRO/1000 NICs not working

dilidolo — Tue, 24 Nov 2009 11:34:10 GMT

I didn't fix the issue, just install ESX 4 U1.

Both issues mentioned above are fixed in U1.

HP Virtual Connect versus HP ProCurve (blade switches) in a c7000?

Paul1 — Mon, 23 Nov 2009 15:12:14 GMT

Sorry, I can't answer your question, because we only use Flex-10. I would say Flex-10 is very nice to have, but do you really need virtualization of MAC-adresses? If you want to virtualize your SAN-environment, i think it's a very great solution and can save really a lot of work for your SAN-Admin. But MAC-adresses in a VMware environment have no special importance so i see no need to virtualize them. But may be there are many future projects which will eventually rely on Flex-10, so i think it's an investment in the future. But at the moment it's much money for only a few advantages. I'm very happy with Flex-10 and it's working perfect so if you have the money, it's a good investment. But otherwise you can also live without it.

VMXNET3 No Performance Data VSphere

Paul1 — Mon, 23 Nov 2009 09:15:20 GMT

Seems to be fixed with the latest Updates from Nov 19, 2009: (VMware ESX 4.0, Patch ESX400-200911201-UG: Updates Core); KB 1014792
•Fixes an issue where networking performance data is missing when the VMNEXT3 adapter is used. The Networking panel is missing in the Performance tab of a virtual machine when a guest is using a VMNEXT Generation 3 adapter. If a virtual machine has a mix of virtual adapters, the Networking panel of the guests not using VMNEXT3 is still displayed. This issue is resolved in this release.

Cisco Nexus 1000V (separate Host?)

lwatta — Fri, 20 Nov 2009 22:56:53 GMT

Yeah we make it out like Control and Packet need to be on their own NIC. In reality you can stick them anywhere. The amount of traffic we send over those networks is small (except packet when you are doing ERSPAN). The concern on our side is that if the network is busy the VSM could lose some heartbeats from the VEM and remove it. Looking at your environment I would stick the control and packet on the pNICs being used for the Service Console. That way all your control traffic is on isolated from VMK and data.

louis

vCenter Linked Mode

Troy Clavell — Wed, 18 Nov 2009 19:43:51 GMT

Linked mode is available if you have vCenter4 at any level.

vSwitch failures?

paintcheck200 — Wed, 18 Nov 2009 18:32:12 GMT

I've also never seen a vSwith failure, but agree that if it did happen, chances are your facing a much bigger problem than your vswitch.

Updated MS network load balancing guidelines for vsphere?

foxy1977 — Wed, 18 Nov 2009 11:36:42 GMT

Hi there,

I am trying to find out if there is an updated version for the microsoft network load balancing guidelines for v4. The only one i can find is this one for v3:

http://www.vmware.com/files/pdf/implmenting_ms_network_load_balancing.pdf

I have previously managed to get this up and running on v3 esxi without any problems. I have managed to get a NLB cluster up and running on two servers but i am experiencing weird network connectivity issues. I just want to check my configuration up against the recommendations/guidelines and rule out any NLB configuration problems.

LinkDiscoveryProtocolConfig error removing physical NIC for migration to dvSwitch

esearle — Tue, 17 Nov 2009 22:31:55 GMT

I was in a crunch when upgrading from 3.5 to 4. so I looked at the event log and discovered that it was complaining of the missing configureation line. It only took a search to discover the command to make changes to that setting. In this case the utility creates the setting if it is missing.

Thanks,

Eli Searle
City of Twin Falls IT Dept.
208-735-7280

From: rjholtz <communities-emailer@vmware.com>
To: <esearle@tfid.org>
Date: 11/17/2009 2:36 PM
Subject: vSphere Networking New message: "LinkDiscoveryProtocolConfig error removing physical NIC for migration to dvSwitch" 1q6BKg-3gQQ-5Xiw
,

A new message was posted in the thread "LinkDiscoveryProtocolConfig error removing physical NIC for migration to dvSwitch":

http://communities.vmware.com/message/1419584#1419584

Author : rjholtz
Profile : http://communities.vmware.com/people/rjholtz
Message:

Multiple VLANs on one vSwitch

jpdicicco — Tue, 17 Nov 2009 22:03:26 GMT

You need to have both VLAN 5 and VLAN 10 trunked between your physical switches. If the links are Cisco, you should just use ISL and either lock it down to your list of VLANs, or allow all VLANs to traverse the link.

JP

Accidentally removed vmnic0 from vSwitch0 - ESX 4.0 & vSphere

norcalwaverider — Tue, 17 Nov 2009 20:05:52 GMT

Thanks for the syntax and pointer. I had to recreate the vswitch0 first and then execute the cmd you provide, then execute esxcfg-vswif -r vswif0, which i found on the link you provided. Thanks a million!

cheers,
norcalwaverider

VLAN Querry

weinstein5 — Tue, 17 Nov 2009 13:59:17 GMT

As gary pointed you can set up a vlan - but you can also use your existing configuration - as long as physical nics on the vswitch plug into a physical switch that sees the 192.168.1.x gateway you chould be able to connect the VM to a portgroup on that vswitch and it will be able to access the approriate network

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

Difference Between Standard vSwitch on ESX3.5 and vSphere

Anton V Zhbankov — Tue, 17 Nov 2009 13:23:10 GMT

Yes, no difference in standard vSwitches between 3.5 and 4.0

---
MCSA, MCTS, VCP, VMware vExpert '2009
http://blog.vadmin.ru

Any software based switch that can be used for testing VLAN in ESX in box scenario?

christoph.wegener — Sun, 15 Nov 2009 05:40:39 GMT

Hi Matt,

GNS3 should work in your scenario. You can use the NM-16ESW module to emulate a switch. The module is missing a lot of functionality but it should be fine for what you want to do.

http://www.blindhog.net/gns3-how-to-build-a-switching-lab/

Christoph Wegener

Problems with vSphere Install

jpdicicco — Fri, 13 Nov 2009 19:46:43 GMT

Glad to help.
I have not found that subtlety documented anywhere, and it goes against how a network person would expect it to work. When I opened a related case to fix my own issue, support was coy and wouldn't provide an explanation. Bottom line is, a vSwitch is not a real switch.

I'm glad you're working!

JP
P.S. Please award points by marking answers correct or helpful when appropriate.

Second service console on separate gateway

beyondvm — Fri, 13 Nov 2009 14:55:03 GMT

Did you try just setting the gateway setting in the GUI?

Can you upload a screen shot of your networking configuration page? That might help us get a better idea of what you are trying to accomplish.

---
If you found any of my comments helpful please consider awarding points for "Correct" or "Helpful". Thanks!!!

www.beyondvm.com

Any way to get an alert when NIC drops to 100MB or disk latency goes above a certain level in vSphere?

badger77 — Fri, 13 Nov 2009 09:50:58 GMT

Actually another way to monitor this would be disk latency, as when the NIC drops the disk latency for the host goes up.

...however, unless I am being a moron, I cannot see a way to monitor disk latency, apart from manually checking performance, there does not seem to be alert counters for this.

Any ideas anyone?

cheers,

Craig.

Virtual NIC type

AndreTheGiant — Sun, 27 Sep 2009 08:08:55 GMT

Available Network Adapters

The following network adapters may be available for your virtual machine, depending on the factors discussed above:

Vlance - An emulated version of the AMD 79C970 PCnet32 LANCE NIC, an older 10 Mbps NIC with drivers available in most 32bit guest operating systems except Windows Vista and later. A virtual machine configured with this network adapter can use its network immediately.

VMXNET - The VMXNET virtual network adapter has no physical counterpart. VMXNET is optimized for performance in a virtual machine. Because operating system vendors do not provide built-in drivers for this card, you must install VMware Tools to have a driver for the VMXNET network adapter available.

Flexible - The Flexible network adapter identifies itself as a Vlance adapter when a virtual machine boots, but initializes itself and functions as either a Vlance or a VMXNET adapter, depending on which driver initializes it. With VMware Tools installed, the VMXNET driver changes the Vlance adapter to the higher performance VMXNET adapter.

E1000 - An emulated version of the Intel 82545EM Gigabit Ethernet NIC, with drivers available in most newer guest operating systems, including Windows XP and later and Linux versions 2.4.19 and later.

VMXNET 2 (Enhanced) - The VMXNET 2 adapter is based on the VMXNET adapter but provides some high-performance features commonly used on modern networks, such as jumbo frames and hardware offloads. This virtual network adapter is available only for some guest operating systems on ESX/ESXi 3.5 and later.
VMXNET 2 is supported only for a limited set of guest operating systems:
- 32 and 64bit versions of Microsoft Windows 2003 (Enterprise and Datacenter Editions). You can use enhanced VMXNET adapters with other versions of the Microsoft Windows 2003 operating system, but a workaround is required to enable the option in VMware Infrastructure (VI) Client or vSphere Client. See Enabling enhanced vmxnet adapters for Microsoft Windows Server 2003 (1007195) if Enhanced VMXNET is not offered as an option.
- 32bit version of Microsoft Windows XP Professional
- 32 and 64bit versions of Red Hat Enterprise Linux 5.0
- 32 and 64bit versions of SUSE Linux Enterprise Server 10
- 64bit versions of Red Hat Enterprise Linux 4.0
- 64bit versions of Ubuntu Linux

VMXNET 3 - The VMXNET 3 adapter is the next generation of a paravirtualized NIC designed for performance, and is not related to VMXNET or VMXNET 2. It offers all the features available in VMXNET 2, and adds several new features like multiqueue support (also known as Receive Side Scaling in Windows), IPv6 offloads, and MSI/MSI-X interrupt delivery.
VMXNET 3 is supported only for virtual machines version 7 and later, with a limited set of guest operating systems:
- 32 and 64bit versions of Microsoft Windows XP and later
- 32 and 64bit versions of Red Hat Enterprise Linux 5.0 and later
- 32 and 64bit versions of SUSE Linux Enterprise Server 10 and later
- 32 and 64bit versions of Asianux 3 and later
- 32 and 64bit versions of Debian 4/Ubuntu 7.04 and later
- 32/64bit versions of Sun Solaris 10 U4 and later

Adapter Caveats

This section discusses some potential problems you might have.

Migrating virtual machines that use enhanced vmxnet
VMXNET 2 is new with ESX 3.5 virtual machines configured to have VMXNET 2 adapters cannot migrate to earlier ESX hosts, even though virtual machines can usually migrate freely between ESX 3.0 and ESX 3.0.x.
If you must migrate a virtual machine between later and earlier hosts, do not choose VMXNET 2.

Upgrading from ESX 2.x to ESX 3.x
When a virtual hardware upgrade operation transforms a virtual machine created on an ESX 2.x host to an ESX 3.x host, Vlance adapters are automatically upgraded to Flexible. In contrast, VMXNET adapters are not upgraded automatically because most or all Linux guest operating system versions do not reliably preserve network settings when a network adapter is replaced. Because the guest operating system thinks a Flexible adapter is still Vlance, it retains the settings in that case. If the upgrade replace a VMXNET adapter with a Flexible adapter, the guest operating system erroneously discards the settings.
After the virtual hardware upgrade, the network adapter is still VMXNET, without the fallback compatibility of the Flexible adapter. Just as on the original earlier host, if VMware Tools is uninstalled on the virtual machine, it cannot access its network adapters.

Adding virtual disks
Adding an existing earlier (ESX 2.x) virtual disk to an ESX 3.x virtual machine results in a de-facto downgrade of that virtual machine to ESX 2.x. If you are using ESX 3.x features, such as enhanced VMXNET or Flexible network adapters, the virtual machine becomes inconsistent. When you add an existing ESX 2.x virtual disk to an ESX 3.x machine, immediately use the Upgrade Virtual Hardware command to restore the virtual machine to the ESX 3 version. This problem does not arise when you add earlier virtual disks to an ESX/ESXi 4.0 virtual machine.
Note: Executing Upgrade Virtual Hardware changes the ESX 2 virtual disk so that it is no longer usable on an ESX 2 virtual machine. Consider making a copy of the disk before you upgrade one of the two copies to ESX 3 format.

Problems with VMware FT
The new vmxnet3 virtual NIC is not compatible with VMware FT, see: http://blog.scottlowe.org/2009/07/05/another-reason-not-to-use-pvscsi-or-vmxnet3/

Performance difference

http://www.vmware.com/pdf/vsp_4_vmxnet3_perf.pdf
http://blog.vmpros.nl/2009/09/22/vmware-performance-evaluation-of-vmxnet3-virtual-network-device/

Reference

Choosing a network adapter for your virtual machine - http://kb.vmware.com/kb/1001805
http://blogs.vmware.com/networking/2009/06/what-vnic-choosing-an-adapter-for-your-vm.html
vmxnet3 - features and use information - tips and tricks
Which NIC for Windows 2008? E1000 or VMXNET 3?

comparison of performance between flexible and vmxnet3 nic

AndreTheGiant — Thu, 12 Nov 2009 19:10:14 GMT

Have a look at:
http://www.vmware.com/pdf/vsp_4_vmxnet3_perf.pdf
http://blog.vmpros.nl/2009/09/22/vmware-performance-evaluation-of-vmxnet3-virtual-network-device/

Andre

this host does not have any virtual machine networks, or you dont haev the permission to access them

storagevmotion — Thu, 12 Nov 2009 17:19:02 GMT

issue resolved until vsphere Updaet 1

VM disconnect from DVS

Bisti — Thu, 12 Nov 2009 14:05:07 GMT

So i thought. I`ll file a SR than and try to reproduce the issue.

Virtual ESX 4 under ESX 4 can't ping out or in

djswarm — Thu, 12 Nov 2009 11:57:59 GMT

Well I was fooling around with it and loaded ESXi 4 and that worked so I blew the ESX 4 vm away and reloaded from scratch, took all the same settings and low and behold, now it works fine. Nothing seems different except for the working/not working. I guess there was a glitch in the original vm somewhere.

vCenter and Nated IP

mgfreaky — Thu, 12 Nov 2009 09:07:49 GMT

Hi,

we have the same problem in our network.

If you click on next link you will find manual how to modified ESX and vCenter for work over NAT.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010652

The very important thing is that you must configure host file of ESX (file where DNS names are mapped with IP addresess). There must be DNS names of every ESX in cluster and vCenter.

So, you must modify each host file of every ESX in cluster:

For vCenter you must write NAT IP address - IP address over wich ESX's see vCenter

For ESX... well, there is problem... :-/

If you map ESX hostname with ESX's physical address HA will work, but clone and copy functions will not work.

If you map ESX hostname with ESX's NAT address (IP address over wich vCenter sees ESX's) clone and copy functions will work, but HA will not work.

We are still trying to solve the problem, but, for now, this is the neares of final solution.

I hope it's not too much complicated for understand.

P.S. I post probleme here: http://communities.vmware.com/thread/240845 and there you can find example scheme.

VMXNET 3 reduces performance

BigHug — Wed, 11 Nov 2009 17:10:14 GMT

What guest os have the issue? We have been using VMXNET3 since vSphere is released. Most of them are upgraded from hardware version 4. We see the performance are improved. Please post back when you get new information.

Isolating VMotion traffic

Sly — Wed, 11 Nov 2009 07:03:12 GMT

As a follow up to this older post, I learned that another good reason to separate VMotion traffic on its own switch and not a VLAN and I wanted to pass on this learning experience for others to consider... and that is that some switches, such as the Cisco 3750 that we are using, do not support enabling Jumbo Frames on individual ports or VLANs and this protocol must be enabled for the entire switch. Since the majority of the traffic on this switch is LAN traffic, it would not be wise for us to turn on Jumbo Frames for the entire switch. As a result our VMotion performance is not as good as it could be.

Netflow in ESX4? what happened?

michael.custance — Mon, 09 Nov 2009 21:30:49 GMT

is this the answer????

http://www.inmon.com/products/virtual-probe/index.php

Looking for Best dvSwitch Solution with 8 pNICs per host

chrisaug — Mon, 09 Nov 2009 17:51:48 GMT

Thanks. I did some more reading and decided to migrate from a single dvSwitch with multiple port groups to multiple dvSwitches with a single port group. Rather than breaking up the port groups and assigning uplinks based on use and traffic I seperated it by dvSwitch.

Thanks for your help.

Machines on distributed and standard v switches not communicating

firestartah — Mon, 09 Nov 2009 13:56:20 GMT

Problem was fixed by changing the load balancing setting to our standard as the default was "Route based on orginating virtual port" which obviously caused a clash/loss of connectivity.

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

Gregg Robertson, VCP3,4 , MCSE, MCSA, MCTS, MCITP

1 vSwitch 2 Vmnic

mcowger — Mon, 09 Nov 2009 00:32:26 GMT

They can coexist (though its not totally best practice - at minimum make them on difference portgroups/VLANs.

--Matt
VCP, vExpert, Unix Geek

Calculate Data Transfer for a VM

mbleske — Sun, 08 Nov 2009 01:00:54 GMT

Our switches (HP Procurve 2848's and 3500's) don't allow metrics at the VLAN level... only at the Port level. This makes tracking on a per VM level (which have individual VLAN's assigned to each) a job for vSphere... hopefully it is possible.

Also, our Senior Network Engineer says that using Netflow for billing purposes isn't really recommended in the networking world anyways...

Any other ideas?

Network performance between two VM's on same ESXi host

athlon_crazy — Sat, 07 Nov 2009 23:31:17 GMT

Could be because of :

iscsi - ip storage based tcp/ip translation overhead
disk I/O - SATA + iscsi combination may contribute into this (based on my actual experience)
network latency - but you should get better if the VM in the same vSwitch, port group or VLAN

vcbMC-1.0.6 Beta
vcbMC-1.0.7 Lite
http://www.no-x.org

Network-Settings vSphere4

hansis — Sat, 07 Nov 2009 08:32:43 GMT

many thanks to all.

hansi

Host only Configuration

AndreTheGiant — Sat, 07 Nov 2009 07:54:45 GMT

As written before use a "isolated" vSwitch without uplink.
For more info about networking in ESX:
http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Andre

Distributed Switch - SPOF ?

jjkrueger — Fri, 06 Nov 2009 15:12:09 GMT

You can look at it this way:

The vCenter piece of the Distributed Virtual Switch is for consistent management across datacenters. You could loosely imagine it as the supervisor module in a physical chassis-based switch.

But the actual virtual switching is done at the ESX host. vCenter will create a hidden standard vSwitch on each host attached to the dvSwitch. In a way, this is like a line card in a physical chassis-based switch.

If vCenter dies (physical, virtual, whatever), the virtual switching plane on each of the hosts remains intact and functional.

My physical analogies aren't completely perfect, I admit, but I think they help with the visualization of the solution.

-jk

Disconnected Service Console after deleting Virtual Nic

rdiphoorn — Fri, 06 Nov 2009 13:18:51 GMT

My situation:

1 vSwitch -> 2 Service Console portgroups

When I remove one of the 2 Service Consoles then the Service Console is only having network connection again when I issue this command on the console: service network restart

Anybody experienced this before? And how to solve it? It's like the Service Console Default Gateway is removed when I delete the second Service Console. (I have hardcoded the gateway in /etc/sysconfig/network)

MAC Address retention required

mk1968 — Fri, 06 Nov 2009 12:19:07 GMT

Does the virtual machine mac address change with an upgrade of VMware tools and virtual hardware within a Windows Server 2003 environment?
I am upgrading from version ESX Server 3.0.1 to version vSphere 4 and NEED to keep the MAC address of one of my Virtual Machines for licensing purposes.
I have had a look at this article however do not want to reserver MAC addresses manually if they will not change with the upgrade.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=9183481

Any advice or experience of wanting to do similar would be appreciated.

What is the limit of users that can connect simuntaniously to vSphere Infrastructure using the vSphere client

Anton V Zhbankov — Thu, 05 Nov 2009 19:58:53 GMT

Concurrent vSphere client connections (32‐bit OS server) 15
Concurrent vSphere client connections (64‐bit OS server) 30

Let them use terminal access to VMs directly, use vSphere client for administrative purposes only.

---
MCSA, MCTS, VCP, VMware vExpert '2009
http://blog.vadmin.ru

NIC Configuration on a Dell m1000?

NYSDHCR — Thu, 05 Nov 2009 14:26:20 GMT

We are planning our vSphere4 install. We now have a Dell m1000, with 5 - m610s (half-height blades - 6 physical connections) and 3 - 710s (full height blades - 12 physical connections), and a Cisco 3130 switch on the back. This would give us 6 pnic connections to each blade. We will be running SC, iSCSI, NFS, Fault Tolerant, and 3 seperate vm networks (server lan, dmz, and a 3-tier). NFS will be 10GB. I would like to be as redundant as possible. We have 52 cpu licenses for Enterprise Plus.

What would be the best possible setups? Bonding 2pnics and creating several vlans? The Dell blade serves up internal connections between blades, as was explained to me by the Dell rep. This could serve Fault Tolerant and vmotion?

vSphere Networking

AWo — Thu, 05 Nov 2009 13:55:01 GMT

If you're going to attach the host to a network with different IP subnets I would take these two NIC's and team them together with a new vSwitch. So you end up with two vSwitches with two NIC's each in a redundant and load balancing configuration.

VCP / VMware vEXPERT 2009
=Due to lack of employees, human beings work here. - Treat them carefully, they are rare.=

Tagged Service Console

ToddStriegel — Wed, 04 Nov 2009 19:01:34 GMT

What about in cases of FCoE? We have each host with two CNAs. The two are in a vSwitch and have multiple port-groups assigned to that vSwitch (Service Console, VMK, VM Networks). In the case where you now only have two connections, what would be the prefered configuration?

VMotion Ports

jeffoutwest — Wed, 04 Nov 2009 18:25:47 GMT

Thanks for the reply. -Jeff

N1000v lost management access

wponder — Wed, 04 Nov 2009 16:11:04 GMT

Thanks for the response. I am still not sure what the root cause was or what exactly the exact resolution was. In either case I was able to recover.

It appears my default NIC vmnic0 used for vm-traffic and vmkernel - management got lost or destroyed in the process of adding the host to the VEM/DVS

Although I tried restoring the standard switch it keep holding the IPaddress of vmnic0 and the physical console would not allow me to configure anything on the management network

I restored it once more and the same thing happened. I waited overnight and this morning the management network was associated to a different NIC that was actually a vmkernel interface for vmotion

Now it would let me configure the NICs again. So, I added another network cable to an available NIC set it as the management interface and gave it a new IPaddress

Once I did that I was able to get connected remotely and start restoring things as they were.

One lesson learned is all my storage traffic is iSCSI / NFS although the vmnic0 interface for management and vmtraffic was toast the " Storage " network was still working and the VMs were powering on. I thought we had vmotioned the VMs off to another host in the cluster but, I guess not. We could not manually add them to a host since they were getting locked by the other host. Could have left the broken host powered off and that would have released them.

IP change is causing problems

robkelley — Wed, 04 Nov 2009 12:55:34 GMT

Just looked at 'vCenter Service Status' and there is an error as follows:

Unable to retrieve health data from http://10.100.101.27:9084/vci/downloads/health.xml

How do I change this IP to the new one?

Creating VLANs using one nic on standard vSwitch on esx4.0

malhareth1982 — Wed, 04 Nov 2009 06:31:52 GMT

I can ping the Default gateway

but I can't ping the Service console IP or its network

Regards

Issues with HA and nexus 1000v switches

lwatta — Tue, 03 Nov 2009 12:20:01 GMT

Chris,

Do you have a VMWare case number or bug id of the problem?

louis

Nexus 1000v Worth it?

tom howarth — Tue, 03 Nov 2009 07:06:41 GMT

This discussion would be better served in the dedicated vSphere Network forum. I am moving it there.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert
VMware Communities User Moderator
Blog: www.planetvm.net
Contributing author on "VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment”. Currently available on roughcuts

Migrating VMs from one cluster (Virtual Switch) to new cluster (vNetwork Distributed Switch)

ChristianWickham — Tue, 03 Nov 2009 00:25:54 GMT

Thanks for your interest, and I am glad I could help someone (even if it is to not do it the same way as me!)

We have the following;
Two blades are BL480G1 and they have 4 onboard NICs at 1Gbps each
Two blades are BL490G6 with FlexNics (8 MAC addresses, so 8 NICs) which I have configured as 4 NICs at 5Gbps each.
I did this to ensure that each blade has the same number of NICs - with a dVS you don't have to do it this way, but I like consistency and neatness.

One important decision we made was to seriously consider the requirement for NIC redundancy. Because these are blades in a single blade chassis, connecting to a single midplane, and the blades have a single chip for all 8 network cards - if there is a failure it is highly unlikely to affect just ONE network card; it would be an all or nothing situation. So, network redundancy needs to be addressed for only two scenarios - an unplugged cable (including at the switch end) and a failed/removed/rebooting Flex-10 module. We decided that NIC redundancy was irrelevant because they are on blades.

So, I created ONE distributed switch, with four port groups.
dvPortGroupGeneral - no VLAN tag, assigned to dvUplink1 and dvUplink2, with dvUplink3 and dvUplink4 as standby adapters
dvPortGroupServers - VLAN tag 130, assigned to dvUplink1 and dvUplink2, with dvUplink3 and dvUplink4 as standby adapters
dvPortGroupDMZ - VLAN 88, assigned to dvUplink3 with dvUplink4 as standby adapter
dvPortGroupManagement - VLAN 135, assigned to dvUplink4 and dvUplink3 as standby adapter

I have one Service Console in the portgroup General, and one in the portgroup Management, and VMKernel is in the Management portgroup.

Our DMZ traffic is extremely low (just email and a few other small websites), and so it does not affect vMotion performance or stability, and we know that in the event that a server is saturating the network for some reason, that we can still manage the virtual environment through another link.

On the Virtual Connect side, I made 3 uplinks and 6 VLANs;
SharedUplink1 contains ports 2, 3 and 4 on Flex-10 module 1, and port 5 on Flex-10 module 2 (this automatically was set as standby)
SharedUplink2 contains ports 2, 3 and 4 on Flex-10 module 2, and port 5 on Flex-10 module 1 (this automatically was set as standby)
SharedUplink3 contains port 6 on Flex-10 module 1 and port 6 on Flex-10 module 2 as standby
vNet-Servers1 is set to VLAN130 and on SharedUplink1
vNet-Servers2 is set to VLAN130 and on SharedUplink2
vNet-General1 is set to VLAN135 and on SharedUplink1 (see details below about "untagged")
vNet-General2 is set to VLAN135 and on SharedUplink2 (see details below about "untagged")
vNet-DMZ is set to VLAN88 and on SharedUplink3
vnet-Management is set to VLAN135 on SharedUplink3

Then - and you can probably work this out by now - I assigned the network cards in each Server Profile as follows;
Port1, LOM:1-a was set to "Multiple Networks" with "Force same VLAN mappings" and vNet-Servers1, vNet-General1 (with untagged checked)
Port2, LOM:2-a was set to "Multiple Networks" with "Force same VLAN mappings" and vNet-Servers2, vNet-General2 (with untagged checked)
Port3, LOM:1-b was set to "Multiple Networks" with "Force same VLAN mappings" and vNet-DMZ, vNet-Management
Port4, LOM:2-b was set to "Multiple Networks" with "Force same VLAN mappings" and vNet-DMZ, vNet-Management

We don't know why, but for non-VLAN traffic, we had to specify in vNet-General that it was on a valid VLAN, and then specify that it was untagged.

I believe that this provides us with the most bandwidth, full coverage for failure of components or lost links, and economy of network cards and cables.

the resource vim.host.port group is in use

dnetz — Mon, 02 Nov 2009 10:10:31 GMT

Glad it helped. Also, don't forget to award points if you found the answers helpful

How to remove a dvSwitch with activate dvUplink?

Testdrive2009 — Mon, 02 Nov 2009 08:12:45 GMT

Thank you,

it worked

Migrated virtual machine has lost network connection

AndreTheGiant — Sun, 01 Nov 2009 15:51:18 GMT

If you converter a machine, the NIC change and (both in Windows and Linux) you loose your old adapter configuration and you have to enter the data again.
To see the old adapter (for example to remove the old configuration) see:
http://www.tech-recipes.com/rx/504/how-to-uninstall-hidden-devices-drivers-and-services/

Andre

Two LAN connection in guest OS

balaonline — Sun, 01 Nov 2009 05:41:40 GMT

HI all, Greeetings,

I have laptop in that i installed VMWARE workstaion. In that i created a TEAM of 4 win2003 servers.

When i applied NAT , all server ping perfectly. But i need two lan connection in particular two servers

for making network connection. because i need to install software that need two NIC In each two server.

I was tried by applying bridge connection but in guest OS showing one LAN connection only. I down t want NAT , i want to use my own ip address.

Please help me to solve this problem. Feel free to query if u have doubts.

Thank you,

vSwitch - pNIC - iSCSI - MPIO

Randles — Fri, 30 Oct 2009 13:30:50 GMT

I'm going to give that a try again then ! I couldn't get it to discover

vSwitch Bug? or User Error: Delete & Recreate Port Groups = Dead vSwitch

Randles — Fri, 30 Oct 2009 13:18:04 GMT

Has anyone else run into a problem like this?

I had two ESX boxes between which I wanted to set up HA and DRS - so I set about reconfiguring ESXBOX-2's vSwitches to match the configuration of ESXBOX-1's exactly. (Now we can forget HA / DRS etc this is just how I arrived at the vSwitch Reconfig)

I had 4 vSwitches and I wanted to move some Port Groups around - for example I wanted to use vSwitch2 for iSCSI which was currently configured on vSwitch3 - so I deleted some port groups and recreated them in the relevant places. I then by error deleted a good port group on vSwitch2; and promptly recreated it with exactly the same configuration as before.

Here's where I ran into a problem;

I was unable to ping the vmkernal / service console I had created on vSwitch2, and I was unable to ping to that subnet from the CLI "Destination Host Unreachable" on my pSwitches I noted that the Links were up / and aggregated but no packets were flowing. I tried refreshing the network from the CLI - nothing. I was completely stumped. I spent quite a lot of time trying to solve the issue, painstakingly checking my Cisco configs and ESX configs only to be 100% sure that configuration of ESXBOX-2s vSwitch2 exactly matched that of ESXBOX-1s vSwitch2 both on the ESX and Cisco side; which was puzzling as ESXBOX-1 was working as expected.

Then I decided to delete the vSwitch and recreate it instantly my problems were gone ! Is this a bug or a feature that others have encountered? Or is it simply user error

2 nics in different VLANs failing over for each other?????

vtek63 — Fri, 30 Oct 2009 02:31:28 GMT

Yes, I was planning on having them on two seperate network segments. right now i'm testing it with vSwitch0 with one NIC (Service Console & vMotion) on one segment and vSwitch1 with one NIC (Backup Service Console & vMotion) on another.

How will ESX know to use vSwitch1 if vSwitch0 goes down?

%EC-SP-5-CANNOT_BUNDLE2: error with ESX4, Catalyst 6509, Dell R710, & link aggregation

itoperationsgrhc — Thu, 29 Oct 2009 23:18:56 GMT

Using KB Articles, 1001938, 1004048, 1003806, & the VMware Virtual Networking Concepts guide. We set up etherchannels on our ESX4 farm using dell R710 servers with 3x Broadcom dual 1GB NIC cards added. 2 of the interfaces are aggregated for the VM Network & the VMkernel. 8 of the interfaces are aggregated as a virtual switchport on a single VLAN.
On the VMware config the VLAN is not set.

The issue is that we are receiving errors on the Catalyst 6509 containing the following error "%EC-SP-5-CANNOT_BUNDLE2:" . The interfaces are being suspended by the IOS because the speed is dropping from 1GB to 100MB. We perform a shut, no shut and the interface reconnects at 1GB.

When these interfaces drop out of the etherchannel we incur random routing misfires, I.e. the physical machine at 172.16.5.75/23 cannot ping the virtual machine at 10.4.32.82/21 but can ping 10.4.32.80/21 & 10.4.32.81/21. the machine at 10.4.32.82/21 cannot ping 172.16.5.78/23 but can ping 172.16.5.81/23 etc. When we do the shut no shut on the interfaces that have the issue; the interfaces are no longer suspended from the port channel the issue disappears.

The virtual interface was registering variable line transmission speeds on the NICs and this is causing the port-channel suspensions. We set all of the NIC interfaces in esx4 to 1000 Full instead of Auto Negotiate, leaving the Cisco 1GB interfaces at the default auto negotiate.

Why does having an interface in a port channel suspended cause this apparent routing error?

Any body else have a similar issue

I spent some time checking DNS, DHCP, WINS and making sure that the routing was working properly

Sample error log:

Oct 23 06:31:44 MST: %EC-SP-5-COMPATIBLE: Gi2/13 is compatible with port-channel members
Oct 23 06:31:44 MST: %EC-SP-5-COMPATIBLE: Gi2/23 is compatible with port-channel members
Oct 23 06:31:44 MST: %EC-SP-5-COMPATIBLE: Gi2/37 is compatible with port-channel members
Oct 23 06:31:44 MST: %EC-SP-5-COMPATIBLE: Gi3/13 is compatible with port-channel members
Oct 23 06:31:45 MST: %EC-SP-5-COMPATIBLE: Gi3/23 is compatible with port-channel members
Oct 23 06:31:45 MST: %EC-SP-5-COMPATIBLE: Gi3/37 is compatible with port-channel members
Oct 23 06:31:49 MST: %EC-SP-5-CANNOT_BUNDLE2: Gi3/35 is not compatible with Gi2/13 and will be suspended (flow control send of Gi3/35 is off, Gi2/13 is on)
Oct 23 06:31:49 MST: %EC-SP-5-CANNOT_BUNDLE2: Gi2/35 is not compatible with Gi2/13 and will be suspended (flow control send of Gi2/35 is off, Gi2/13 is on)
Oct 23 06:33:54 MST: %EC-SP-5-COMPATIBLE: Gi2/35 is compatible with port-channel members
Oct 23 06:33:53 MST: %EC-SP-5-COMPATIBLE: Gi3/35 is compatible with port-channel members
Oct 23 06:34:10 MST: %EC-SP-5-CANNOT_BUNDLE2: Gi2/23 is not compatible with Gi2/13 and will be suspended (speed of Gi2/23 is 100M, Gi2/13 is 1000M)
000290: Oct 29 14:55:22 MST: %SYS-5-CONFIG_I: Configured from console by admin on vty0

Lost Management connection

bolsen — Thu, 29 Oct 2009 17:30:57 GMT

Assuming you've rebooted, check for physical connectivity lights and switch port settings.

82571EB Full/Half duplex issues

callee — Thu, 29 Oct 2009 13:46:25 GMT

Hello everyone,

I have a bit of a peculiar issue after freshly installing a bunch of vSphere hosts (with all available patches applied).

After each reboot the interfaces which are supposed to be set at 100Mbit/full reverts to 100Mbit/half instead. Switches are configured to be 100Mbit full and not auto-neg. And this issue only seems to apply to my pci-e controllers. The onboard NICS are fine (NetXtreme II BCM5708, hp dl380 g5).

Just to be sure I confirmed the settings in esx.conf:

/net/pnic/child0002/duplex = "full"
/net/pnic/child0002/mac = "00:15:17:33:fa:c8"
/net/pnic/child0002/name = "vmnic2"
/net/pnic/child0002/speed = "100"
/net/pnic/child0002/virtualMac = "00:50:56:53:fa:c8"

but running esxcfg-nics -l gives me the following output

vmnic2 10:00.00 e1000e Up 100Mbps Half 00:15:17:33:fa:c8 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

GUI also displays the same. Setting the nics to 100Mbit/full during operation works fine and continues to run this way until I reboot the host again.

I could just write a script to reconfigure the nics post boot, but surely this shouldn't be needed.

Anyone with any brilliant ideas?

Thanks.

Best Practice in setting up vNetwork Distributed Switches

kopper27 — Wed, 28 Oct 2009 21:31:44 GMT

+"So far my understanding is that vDS is great but because it's managed
on the vCenter layer it can be a big issue. Once you lose vCenter you
cannot control or configure your network component. Is what I'm saying
correct? Can someone confirm this for me. Therefore it makes a lot of
sense to go with the hybrid configuration (vSS and vDS) and the Server
Console/Management console be set as a vSS that way you have access to
any of your esx host from the vSphere client when you lose you vCenter.
"+

yes that's right that negative point on DVS since it is handled by the Vcenter Server...so it is like a new layer manage by the Vcenter that being said if your vCenter fails - either physical o Virtual - and you don't have a backup vCenter you won't be able to manage your your VDS now imaging that you setup the Service Console/Vkernel in that VDS configuration... that would not be any good for any ESX admin.

for instance check this guy's issue with DVS and Service Console mixed

http://communities.vmware.com/thread/218879

help with a dvSwitch setup

jpd4save9 — Wed, 28 Oct 2009 15:20:38 GMT

Really should have posted eariler but have been distracted by hostd being broke.

Well I can back monday and it just started working.

The 3Com must have just taken forever to get itself setup right.

Not much else to say.

VM are not able to ping with Gateway through another NIC (except console nic).

habibalby — Tue, 27 Oct 2009 15:55:19 GMT

Hello,

Make sure the vLAN Trunking is correct within the pSwitches

Make sure the vLAN ID within the PortGroup is associated with the Physical Switch vLAN ID. "Put the vLAN ID in the PortGroup"

Also, check out the following;

Host1

vmnic0 <--------> vSwitch0 <-----------> pSwitch1 Port1

Host2

vmnic0 <--------> vSwitch0 <-----------> pSwitch2 Port1

And the same goes with the other nics.

Best Regards,

Hussain Al Sayed

If you find this information useful, please award points for "correct" or "helpful".

How can I change vswif from a dVS to another and associate it with a vmnic (from CLI) ?

jpdicicco — Tue, 27 Oct 2009 15:29:56 GMT

You can do this through the VIC.

Add a 2nd SC port/nic on the new dvSwitch, connect your VIC via that interface (keeping IP routing in mind), and remove the old SC port/nic.

does network communication among guests under same ESXi host go to physical/real network???

Kallex — Mon, 26 Oct 2009 18:18:20 GMT

Hi!

What would be the expected performance of a "completely virtual" network?

We are planning to provide virtualized NFS share for two separate (local and remote) ESX boxes.

The gotcha is that we're looking to get really high performance NFS drive that is using ramdisk mount below the surface. Now naturally the remote ESX will get glan capped, but the local would preferably have higher throughput. We do have already disk based virtual NFS that's actually providing storage point for the very same host that runs the VM itself (this clause is just before someone says we can't provide NFS mount to the host from its own guest).

Br,

Kalle

Delegated networks with a dvSwitch

purduecjs — Mon, 26 Oct 2009 16:12:28 GMT

As a follow-on to my previous post related to Department role/permissions problem solved! Virtual Center vCenter , I thought it would be a good idea to elaborate on the new permissions available for datastores and networking. However, I'm having a bit of a problem with delegated networking - has anyone else attempted this?

Assume we have an external dvSwitch that includes multiple VLANs (VLAN 10, VLAN 11, VLAN 12, VLAN 13, VLAN 14) and I have two departments : department A should have access to VLAN 10, and 11 ... and department B should have access to VLAN 12, 13, and 14.

So I create a role called "Network Manager" with the permission of:

Network -> Assign network

and I assign this role for Department A to VLAN 10 and VLAN 11. I assign this role for Department B to VLAN 12, 13, and 14.

I also have a similar permission set on the VM (or folder containing the VM) for "Network -> Assign network".

So I login as Department A, 'edit settings' for a VM, add device - enet adapter, and I can choose among the VLANs that I have given perms on ... specifically VLAN 10 and 11. I finish adding the device and click ok to reconfigure the machine, then 'edit settings' again to change the VLAN from 10 to 11 and I get the error:

"This host does not have any virtual machine networks, or you don't have the permission to access them."

A work-around that I've discovered is to add the dvSwitch in question to a folder. Then on this folder add the "Read only" role for all of your deparmental groups (Department A and Department B, from my example). When they go to "Edit settings", behavior is as expected and they will only be shown those networks that have been granted "Network -> Assign network"

I've filed a request with support and they are looking into it, but I wanted to reach out to see if anyone has thoughts on a better way to delegate or if anyone else can replicate this issue.

Thanks very much,

Cameron J. Smith
System Administrator, Purdue University

Slow network and packets drops when one VM (vmxnet3) is under heavy traffic

Docent — Mon, 26 Oct 2009 11:12:44 GMT

Hi,

I have an ESXi 4 installation with three VMs running on top and two physical network adapters. One VM is used as a router machine for my network, it's Debian and has open-vm-tools installed. It has two vmxnet3 network adapters. One adapter is connected to a dedicated vSwitch ("External") and the other is connected to the second vSwitch ("Internal") with all other VMs. Virtual OS is configured with VLANs, so ESXi passes all VLAN IDs on the External vSwitch. On the other ESXi host there is an identical VM with Debian and the same configuration. They're both working in a failover cluster using linux-ha.

There is no problem when the machine is passive, e.g. not routing any traffic. However there is an issue with the active node: it's nearly impossible to connect to the services (web, remote desktop, etc.) hosted on the other VMs running on top of the host with active router node. The connection frequently drops and the file transfer via HTTP is just a few KB/s. What is interesting, the ESXi host is affected itself as the download of vSphere Client goes around 5 KB/s as well. When I shutdown the primary router node (thus causing failover to another), everything goes back to normal on that host but the problem moves to the secondary node. Linux itself is unaffected and it can handle heavy traffic of several hundreds of mbit/s.

The problem is gone when I switch vmxnet3 adapters on Linux to e1000. However e1000 doesn't work well with heavy traffic like >100-200 mbit/s. So I'm stuck with vmxnet3 and would really appreciate any suggestions.

Thanks,
Docent

configuring diff. IP space

Anton V Zhbankov — Mon, 26 Oct 2009 08:35:12 GMT

That's actually not VMware related, just a general networking.

Depending on your network/VMware configuration you can set up isolated vSwitch for 192.168.x.x and set up VM to work as router. Or you can set up VLANs, or ... many possibilities.

---
MCSA, MCTS, VCP, VMware vExpert '2009
http://blog.vadmin.ru

Network drivers for WinPE

AndreTheGiant — Sun, 25 Oct 2009 12:54:44 GMT

Have you tried with a generic Intel Pro 1000 driver for e1000 and a VLANCE (AMDPro card) for flexible (when is it used without VMware Tools)?
See also:
http://sanbarrow.com/phpBB2/viewtopic.php?t=837&sid=f58d8d5cffb6f6409b09ded7eb6a6145

Andre

Question on best practice for NIC cards on ESX host.

AndreTheGiant — Sun, 25 Oct 2009 09:31:49 GMT

Your setup also will depend on VLAN usage or not.
You have a "single" switch (cause you use stacking) so a VLAN partitioning could be a good idea.

Andre

ESX4i - 2 servers and 1 HP Switch - VLAN / Trunking ???

athlon_crazy — Sat, 24 Oct 2009 21:36:34 GMT

Yes you can. For VST method:

Pro-Curve

Just create VLAN 90 in your Pro-Curve 1800G
Select two ports each connected to NICS of your ESXi hosts as member to VLAN 90
Choose tag=ALL for your VLAN config

ESX

Create port group with VLAN ID 90

vcbMC-1.0.6 Beta
vcbMC-1.0.7 Lite
http://www.no-x.org

To etherchannel or not?

vanak — Fri, 23 Oct 2009 15:12:48 GMT

You should make aliases on your netapp vif, in order to get several pairs of ip involved in your network communications. If you've have only one IP referring to one datastore (NFS for your case), etherchannel is not very usefull. It's great with all vSwitch & vmkernel configured with "route based on ip hash", you network switch well configured and several different Ip connections to your nfs datastores.

Here they are several usefull links, and for sur they will be more than helpfull for what you want to plan

Cheers.

http://blog.scottlowe.org/2008/07/16/understanding-nic-utilization-in-vmware-esx/

http://communities.netapp.com/blogs/ethernetstorageguy/2009/04/04/multimode-vif-survival-guide ( you may need a "now account" on netapp for this one)

2003 R2 x64 vNIC - Flexible vs. E1000?

jpdicicco — Fri, 23 Oct 2009 14:33:24 GMT

It's different hardware, so the OS sees it as different NICs. Be prepared to reconfigure the IP settings on the new NIC.

If you run into a case of Windows complaining of duplicate IPs, then you need to manually delete the old NIC...

Open a command prompt and run: set devmgr_show_nonpresent_devices=1 && devmgmt.msc
In the View menu, select "Show hidden devices"
Go to the Network Adapters list
Right-click and uninstall the old NIC

Problems with network traffic, using VLANs and trunks.

FredericPerrin — Fri, 23 Oct 2009 12:49:38 GMT

If I understand correctly, the VM's (untagged) packets enter trunk 1, where they are tagged with vlan 20, before going to port 19 and reach the rest of your subnet 192.168.0/24. If that is correct, then when the answer to this packet reach the VM, they still have the tag VLAN20, so your VM doesn't know how to handle them. You should be able to see them with a tcpdump, though I'm not entirely sure.

Configure your VM's portgroup to be in VLAN20. Than, the portgroup will tag / untag the packets before sending them to the network / to the VM.

VMWare and VLAN w/virtual firewall

Anton V Zhbankov — Thu, 22 Oct 2009 21:13:12 GMT

vShield supports VLANs, so not a problem.

---
MCSA, MCTS, VCP, VMware vExpert '2009
http://blog.vadmin.ru

Test Environment

NTurnbull — Thu, 22 Oct 2009 15:56:05 GMT

If your sure that the physical switches can't communicate with each other then as long as you only connect each vm to one vDS then no, they wouldn't be able to communicate. The only way they could is if you assign another vnic to one of the vm's and attach that vnic to the other vDS aswell

Thanks,
Neil

Network performance suddenly slowed to a halt

itbegary — Thu, 22 Oct 2009 14:06:26 GMT

I have 3 servers, 2 are Dell 1425's running ESXi 3 (box A and B), another is a SuperMicro running ESXi 4 (box C). Everything was running fine until about 4 days ago. I noticed that one of the guests on box A was getting a mind blowing 40kb/s file transfer rate when doing a yum update against a repository on box B. Testing showed that any instance on box B or C can access the repo just fine and get various speeds well above the 5MB range when doing file transfers.

Box A had both Windows and Linux guests on it. It doesn't seem to matter which guest OS is installed, I still get this slowness. Sometimes on the Windows box, it's fine. Other times, it's not. I suspect that the same is true for the Linux boxes.

The only change I made about 10 days ago was to add a virtual switch on a seperate VLAN and put a single server on that one. I created this VLAN on all ESXi hosts though (in the same way with the same VLAN ID's). I would suspect that if this change is the root cause that I would be seeing this problem across the board. scp'ing a file from box B to/from box A is about 5mb/s so the physical network seems fine.

Realtime:
CPU on the affected host is about 30% (10 guests), memory is about 73% (6GB of 8GB allocated and used), disk is averaging 100kb/sec, network 17kb/sec.

Any ideas on how to approach this problem?

Move physical adapters to DVS

mcvosi — Wed, 21 Oct 2009 15:48:55 GMT

Got it.

Thanks.

Multi-gigabit uplinks to VMs with Flex-10 LACP

jpdicicco — Wed, 21 Oct 2009 15:28:59 GMT

Sorry for not being more clear. I'm asking what type of NIC you are using your guest OS (i.e. VMXNET, e1000, etc.). The virtual h/w can determine what NIC speed shows to the guest OS. Am I correct in understanding that your issue is the guest OS connection speed and not the vmnic/pnic speed to the VC module?

For the VC config, the key is to not use VC for redundancy directly, but to use VMware's failover capabilities. I'll try to explain it here with the why's, but there are many relevant parts:

Physical setup-

We have 8 interconnects: 1,2,5,6,7,8 are 1/10G ethernet; 3,4 are FC
Our hosts have the 2x1G LOM interfaces, and a quad port card in Mez 2, with 1 i/f connecting to each of bays 5-8

I have 1 Shared Uplink Set (SUS) for each VC module (since we cannot have active ports in 1 SUS from 2 different modules; I'm not sure how this breaks out on Flex-10 modules). This is the key to keeping all ports active in our config, that we have 1 SUS per group of uplinks that can be trunked as active ports.
On the switch side, we have a Cisco Cat6500, using etherchannel for aggregation. An example config is attached.

In VMware networking config, we have 1 LOM vmnic (vmnic0) and 1 vmnic from the mez. card vmnic (vmnic2) assigned to our standard vSwitch. This switch has our SC and VMotion connections with load-balancing set to active/standby as follows: SC=vmnic0/vmnic2; VMotion=vmnic2/vmnic0. So under normal conditions, each port has a dedicated vmnic (best practice).

The remaining vmnics are on a dvSwitch. All port groups use all vmnics active/active, with Port ID load balancing.

Summary / rules governing our setup-

1 SUS per interface or group of interfaces that can be active/active
For each SUS that will be on a given vSwitch, every VLAN ID must be in each SUS (example in attached file "SUS networks list.png")
VMware load balancing CANNOT be set to IP hash. IP-hash will cause MAC thrashing on your physical switch!

Upsides-

All VC interfaces are active and usable at any given time
Load sharing occurs based on vSwitch "load balancing" setting of Port ID

Downsides-

The stacking links go unused (not such a bid deal)
When all uplinks in a module go offline, our vmnic is set to link down (based on Smart Link being enabled). Thus, we lose 1 uplink worth of bandwidth to the VC modules as well.
Lots of manual setup in VC.

Problem with VM on DVS

mcvosi — Tue, 20 Oct 2009 20:12:17 GMT

A resolution to this... I was able to fix this by installing a newer version of IOS on my L3 switch.

intermittent network loss for single clients on shared vSwitch

macallan — Tue, 20 Oct 2009 13:52:42 GMT

This looked like a Cisco {font:Arial}{size:10pt}{font:Arial}{size:10pt}port security problem, but the network guy reported that this server's switch connections had two ports configured for {font}{font}link aggregation. The solution is to put these two NICs onto a vSwitch configured for NIC Teaming with Load Balancing: Route based on ip hash. Now the two NICs work correctly as a team, and there aren't random failures of VM connectivity.

NLB - Two VMs can't ping eachother when on seperate hosts

jpdicicco — Tue, 20 Oct 2009 13:33:26 GMT

You're most welcome. The reason you couldn't do a `route change` is that it is an automatic route. Each interface is given a route to its local network based on its IP/Mask combination. You can't change those, since they are based on the IP configuration of the interface. So, you had to add one that overrides it.

Another tactic would be to add a route specifically for the mgmt interfaces, a la `route add 172.18.18.52 255.255.255.255 IF 172.18.18.51`. Whatever works for your environment, is how you should do it.

Keep in mind, that any additional management traffic that you have (snmp, rdp, backups, etc.) that has to go to one interface or the other may require additional adjustments to your routing.

Understanding vSphere™ IP Pools

martinamdisen — Tue, 20 Oct 2009 12:58:03 GMT

Hi Nicholas

We have some blog posts on vApps here:
http://blogs.vmware.com/vApp/

The first one, called "OVF and vApps in Action" shows an application of IP pools. Hopefully we can provide a video showing how to setup IP pools in the future.

Thanks,
Martin

ESX 3.5 and 4.0 LACP support

jpdicicco — Mon, 19 Oct 2009 19:30:30 GMT

I stand corrected. I was, as I am sure they are, using the term LACP interchangably with 802.3ad (now 802.1AX-2008). The correct usage would be LAG for Link Aggregation. LACP refers specifically to the negotiation protocol. They should update their documentation or their code.

ESXi 4 - I have 2 vmkernels on two separate networks, why only allowed 1 gateway?

jpdicicco — Mon, 19 Oct 2009 17:20:17 GMT

This is standard networking behaviour for IP. You can only have 1 default route, which is what this setting represents. You have 2 options to set IP routing for iSCSI:

1. Setup another VMK NIC on the subnet of your target. This should cause packets to automatically route out that interface.

2. On the service console, use the esxcfg-route command to build a VMk routing table to your liking.

Setting up VMDq on ESX 4.0

VMentor — Mon, 19 Oct 2009 16:55:36 GMT

Hi,

I have the same question. I'm currently playing with a Dell PowerEdge R710 that has 2x Intel Gigabit ET quad port NICs (Intel 82576 controller) installed and I didn't find the trick to enable VMDq on ESX 4.0 and ESXi 4.0. I've set the options as described in the document for ESX 3.5 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009010) but nothing happens. When I run ethtool -S vmnic* I can see only one RX-queue.

Is there anybody outside who is running VMDq on ESX 4.0?

regards, Simon

Easy way to remove a vDS?

jpdicicco — Mon, 19 Oct 2009 16:36:21 GMT

Go to Inventory-> Networking and find the Hosts tab for the switch

Select the host to remove from the dvSwitch and press the delete key

---or---

Right-click the host and select "Remove from Distributed Virtual Switch"

ESX host disconnects from vCenter when adding nic to form team on switch with service console

jpdicicco — Mon, 19 Oct 2009 16:30:02 GMT

I've just recently setup a c7000 with Virtual Connect modules. Are you using VC modules?

vDS network problem

jpdicicco — Mon, 19 Oct 2009 15:58:05 GMT

Sounds like you are using Cisco Etherchannel. For load-balancing between an ESX host and a Cisco switch using Etherchannel, you MUST use IP hashing for your load-balancing algorithm.

vSphere and Cisco Nexus

lwatta — Mon, 19 Oct 2009 13:24:10 GMT

I've seen it happen once or twice. Usually (not always) its something on the physical switch the ESX hosts are connected to. Let me know what you find out.

louis

Strange issue with ESXi 4.0 and routing

kenwardc — Sat, 17 Oct 2009 17:38:19 GMT

Hi Folks

I'm new to VMWare. I have installed ESXI 4.0 on a DELL 2950 server with two NICs. When I originally installed the software and started out on my learning curve, I did it at home and used my ADSL router and ADSL line as the network. The only machine on the network was the DELL 2950 and the VMs I created inside it. Everything worked 100% to and from everywhere. I could see the DNS servers and surf the web from any of the created VMs (combination of Windows 2008 Server and CentOS 5.x) so just assumed everything was set up correctly.

Now, I've moved the box to our rack space at the datacenter and the people there have given us a brand new network range to use with the machine, because basically we weren't given enough IP addresses to grow to the number being used on the old range, let's call it 12.13.14.x. The new IP range has been put into our Netscreen firewall routing and the Management Console was used to change the network IP addresses, subnet mask and gateway to the new address range before we moved the DELL unit to the rack.

The rack has an HP Procurve network switch as the interface between the Datacenter switch (the outside world) and our servers. So... it's switch, firewall, servers.

The address range is: 2.3.4.192 - 2.3.4.255 with .193 being the first available address and .253 the last. The .192 and .255 IPs are used for broadcast addresses and the gateway address set to .254. The subnet is /26 or .192

The host is set up as follows:
NIC number 1: Set up with address 2.3.4.193, subnet mask .192 and gateway of .254
NIC number 2: Set up with address 2.3.4.194, subnet mask .192 and gateway of .254

On advice from the datacenter support team, I separated the VMkernel Port (using vSwitch 0) and made it the managementnetwork. It is attached to physical adaptor vmnic0 and the IP address is set to and appears as 2.3.4.193. As I can see the machine from remote, I'm assuming this is working OK. I can attach to the host using the vSphere client and manage everything as expected.

The VMs are all attached to the second network, called vSwitch1 (Virtual Machine Port Group) and appears to be using adaptor vmnic1. There is no mention of IP address so I have no idea whether this network is using the .194 address as above or not.

Just to throw you all a little curved ball, looking at the "Observed IP Ranges" in network properties from the vSphere client, it says that the observed IP ranges (on both NICs) are 12.13.14.x and there is no mention at all of the new range i.e. 2.3.4.x

Here's the problem. Although I can see the host from the Internet, I cannot see any of the VMs contained inside it from the Internet. If I go to the host using vSphere Client, then choose a VM, open the console, login and open a terminal window, I can ping the host but I cannot ping anything outside the 12.13.14.x range and I cannot ping any of the other Vms. Here's an example of what's happening:

Our DNS servers are both inside the 12.13.14.x range, on 12.13.14.65 and 12.13.14.42

If I ping "google.co.uk" I see an instant lookup on DNS, the IP is resolved correctly, but then there is nothing. No replies.

If I ping ourdomain.co.uk, which is hosted within our 12.13.14.x range, I get the same instant lookup and then lots of ping replies from that host.

If I ping the VM on 2.3.4.196 from the VM on 2.3.4.195 I get absolutely nothing so there is a disconnect there, too.

From where I'm sitting and taking into account my inexperience with VMWare, it seems as though the "Observed IP Ranges" are telling the host to only use that range. How do I fix this issue? Any help would be gratefully accepted.

Regards and thanks
Chris

Intel 10Gbe AF DA (82598EB) dual-port adapter experience?

max.inglis — Sun, 18 Oct 2009 10:06:09 GMT

I have an open case with Intel and Vmware about this still, no resoultion yet, but they're working on it.

http://ITangst.blogspot.com

Network segments & virtual switches

AndreTheGiant — Sun, 18 Oct 2009 05:10:31 GMT

You NIC can be connect only to one vSwitch.
So is not possible.

See also:
http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Andre

vmxnet3 - features and use information - tips and tricks

joshuatownsend — Fri, 16 Oct 2009 22:45:51 GMT

Good thread, iben.

One thing to point out - the set devmgr_show_nonpresent_devices=1 command does not work with Windows 2008.

Instead, you have to set an environmental variable of "devmgr_show_nonpresent_devices" with a value of "1". Then you can open device manager and click 'View | show hidden devices' to see the ghost NIC.

Josh

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".
Please visit http://vmtoday.com for News, Views and Virtualization How-To's

What does VLAN off-loading is doing?

RussH — Fri, 16 Oct 2009 15:19:22 GMT

Sorry for resurrecting an old thread - but I too was wondering what the term meant and had a bit of a look into it - I believe this functionality in vmxnet3 allows adding VLAN tags within the guest OS (on the NIC driver) itself.....

Cheers

How to define subnets

amvmware — Fri, 16 Oct 2009 15:01:12 GMT

Can you ping the default gateway of each of these subnets.

Have you created these subnets to only support your virtual environment or do you have other physical servers using these subnets.

If the 192.168.3.x subnet has only been created to support the vswitch then you need to be able to physically route to it. How you do it depends on your environment - layer 3 switching or introduce a router to do the task.

network design suggestions

milos77 — Fri, 16 Oct 2009 14:02:29 GMT

First of all i suggest you to purchase 2 multi-nic card with 4 interface and to devide each service on different card. (This for fault tollerance)

I think you need:

1 Group of 2 interface for Service Console and Vmotion

1 Group of 2 intercace for Iscsi (but EQP6000 has 4 intercace for each controller so you can consider to upgrade to 4 interface on each node for iScsi)

1 Group of 2 intercace for vmNetwork but this depend on how many virtual machine you run on each node

1 Group of 2 interface for vmNetwork for DMZ uses (I suggest you to phisically separate DMZ and LAN traffic)

Remember that is important to use interface card with TOE Capabilities.

Restricting Access to vNetwork Distributed Switch Port Groups?

CayceWill — Fri, 16 Oct 2009 12:52:30 GMT

Argh - the permissions tab.