Help troubleshooting intermittent vm network connectivity

[boffman]

Hi,

I think I may have a network issue with some ESXi 4.1 hosts. The Windows 2008R2 VMs that run on that farm are experiencing unexplained disconnections to a SQL server on another part of the network. Also, in the event log, I am seeing this messsage a couple of times a day:

Log Name:      System
Source:        Microsoft-Windows-DNS-Client
Event ID:      1014
Level:         Warning
User:          NETWORK SERVICE
Computer:      DCXXX.YYY.com.au
Description:
Name resolution for the name YYY.com timed out after none of the configured DNS servers responded.

One possible explanation is that there is an issue with the network on the hosts that means connectivity is lost but the virtual adaptors stay up.

Could someone please give me an idea of what to look for (and which logs to look in as well) in the ESXi logs. I am for example seeing this every few minutes in the hostd.log - is it normal?

[2011-06-08 03:01:08.321 32EE3B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
[2011-06-08 03:02:02.000 331C1B90 verbose 'DvsManager'] PersistAllDvsInfo called
[2011-06-08 03:02:02.455 32E81B90 verbose 'DvsTracker'] FetchSwitches: added 1 items
[2011-06-08 03:02:02.455 32E81B90 verbose 'DvsTracker'] FetchDVPortgroups: added 3 items
[2011-06-08 03:02:39.007 331C1B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
[2011-06-08 03:03:36.375 328CBB90 verbose 'ha-license-manager' opID=HB-host-167@58586-a1] Load: Loading existing file: /etc/vmware/license.cfg
[2011-06-08 03:03:36.392 328CBB90 verbose 'App' opID=HB-host-167@58586-a1] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 4.0" (lastError=0, desc.IsValid:Yes)
[2011-06-08 03:03:36.406 328CBB90 verbose 'Locale' opID=HB-host-167@58586-a1] Default resource used for 'LicenseManager.LicenseInfo.dpvmotion.label' expected in module 'default'.
[2011-06-08 03:03:36.406 328CBB90 verbose 'Locale' opID=HB-host-167@58586-a1] Default resource used for 'LicenseManager.LicenseInfo.vaai.label' expected in module 'default'.
[2011-06-08 03:03:56.534 32EE3B90 verbose 'Proxysvc Req39622'] New proxy client SSL(TCP(local=10.48.198.5:44585, peer=10.48.198.8:443))
[2011-06-08 03:04:09.698 32EE3B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
[2011-06-08 03:04:29.960 32EE3B90 verbose 'Proxysvc Req39623'] New proxy client TCP(local=127.0.0.1:52002, peer=127.0.0.1:80)
[2011-06-08 03:04:29.962 32E81B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x33768b20] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
[2011-06-08 03:04:29.962 32E81B90 verbose 'Vmomi'] Arg version:
"1974"
[2011-06-08 03:04:29.962 32E81B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
[2011-06-08 03:04:29.962 32E81B90 info 'Vmomi'] Result:
(vmodl.fault.RequestCanceled) {
   dynamicType = <unset>, 
   faultCause = (vmodl.MethodFault) null, 
   msg = "", 
}
[2011-06-08 03:04:29.963 34040B90 error 'App'] Failed to read header on stream TCP(local=127.0.0.1:49849, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

I have attached the hostd.log file in its entirety in case it is usefull.

[idle-jam]

how about changing virtual adapter type to E1000 or VMXnet3? and see if the same symptom still occurs?

[boffman]

Could do, but I'm not finding anything logged in Windows (other than that mentioned above) to indicate a network problem. If the virtual adapter type or driver was the issue wouldn't I see something in the event log?

[idle-jam]

the windows event log will only tell you the lost of network connectivity and anything from the OS level. i'm suspecting it's actually from the VM level or the hypervisor that is causing this.

[boffman]

so does the changing the adaptor have an effect at the OS or hypervisor level?

[idle-jam]

i have certain implementation that diff. type of adapter type does make difference (we could drill to the root cause later..)

[boffman]

OK - will give it a go tomorrow. We have a few test servers I can try this on. Many thanks.

[muguran]

Did it solve your problem?