Strange Readings From esxtop DAVG KAVG

[brandoncam]

I have been scratching my head over this and show has Dell and vmware Support. When I execute esxtop u I get - 18446744073709552.00 it fluctuates between that and another number and through multiple LUNS.  I have updated all drivers/firmware including SAN\Server\Vmware..... Has anyone ever seen anything like this? I am running ESXI 5.5.  I am not noticing any latency to my storage unit " according to Dell Headquarters  ". However I am noticing some vcenter latency loading up vcenter via web browser and the Client.

The vmware tech stated that it was a storage unit issue so I contacted dell and they went through the logs with a fine tooth comb. We had my SAN switches inspected my PS6100E and my HP2920 Network Switches everything checks out to be working fine.

I am thinking it is a bug in the vmware OS.

Has anyone seen this before or had this same issue? I just want to confirm this isnt a issue that will cause a big problem down there road!

[NealeC]

Hi Brandon,

Have you tried disabling (if possible) vaai? as it has been known to skew davg/kavg values before.

VMware KB: Abnormal DAVG and KAVG values observed during VAAI operations

Here's how to disable VAAI (presuming it's something you can test before you go into production)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=103366...

Regards

Chris

[Peat]

I'm seeing the exact same thing.  I found this thread by googling 18446744073709552.00, which is the number I'm seeing under DAVG/cmd.

It's occurring on ESXi 5.5 2302651 running on a Dell R710 host using Brocade FC switches and an EMC VNX 5700 san.  I don't see any real world issues anywhere.  As you, I suspect a vmware bug.

Peter

[brandoncam]

I wonder if it is a Dell issue? I am using a R720 with N3024 ISCSI Switches. It makes me wonder if anyone with HP servers are having the same issue?

There has been no performance decrease from what I see, I was just concerned that it was a sign of a bigger issue.

[brandoncam]

Thank you for your response, I attempted that solution and it didnt seem to fix the problem.

[brandoncam]

what NIC cards are you using in your R710?

[Peat]

I am seeing this activity on fibre channel data stores.  We're using Qlogic 2562 Dual Channel 8Gb Optical Fiber Channel HBA PCIe.  For NICS we're using the embedded Broadcom NetXtreme II BCM5709 and Intel 82576 based quad adapters.

Peter

[rtarantola]

Same exact issue with crazy davg/kavg values with Dell Switches,Dell San and HP hosts.Thank god for this post because dell and vmware are useless.Thanks guys!!

[waynej]

Same thing here on Dell R710 (Intel 10Gb NICs), R720's (BC 10Gb NICs), Cisco Nexus 5548 switches and VNX5300's.

[rtarantola]

I can't believe the "Level 3 Techs" at VmWare couldnt figure this out and we all could.Is there any way we could all to push for this bug fix in the new build?

[NealeC]

Hmm the consistent component I can see is the Dell R710s.

Is it not more likely that the driver for the HBAs in there isn't getting returned the same data when it makes the same API call it does on all other HBAs and gets the right stats back?

(although I'll wait for someone to say R710s are in the HCL 🙂 )

Chris

[brandoncam]

if you made it to level 3 VMware support your lucky. I had Vmware Blame it on Dell Storage Unit and said there is nothing they can do. That is all they would do for me!

Dell Rep I have been talking with has brought everyone in on my case and Has been very helpful!  EQL- Networking- Dell VMware Unit

What we have done

Firmware/Drivers on the following

Update NIC ..

Updated SAN To current EQL drivers

Updated HP Switches

Updated Dell ISCSI Switches

I am only running 1GB links on my NICS

I did get a email from the tech who said there are new patches as of few days ago that might fix this issue *sign* ill keep you guys updated!!

I would say it would be the Dell 720's or 710 but there is a set of HP host in this post who is experiencing the same issue.

If anyone comes up with a solution PLEASE let me know.  I will do these updates/patches on the host and let you guys know what is going on!

[rtarantola]

I have this situation occurring on all of my hosts

HP Proliant dl380 g5

HP Proliant dl380 g6

HP Proliant dl380 g7

HP Proliant dl380 g8 so I doubt this is just an issue with Dell 710 and 720's.Vmware was trying to convince me that my "outdated" nic drivers were the issue and that as far as I got with them.They have been unresponsive to my claims that this is a software bug.They claimed the person who helped was a "senior tech or level 3"  but I doubt he was.My case has been going on for two months.VmWare should be ashamed of themselves!

[Peat]

I have checked and I am seeing this on Dell R710, Dell R820 and on the Dell R730 I just built.

I am going to open a ticket with VMware support to add another voice to the list with them.

Peter

[Mackan_Sweden]

I'm also having the same issue.

I've just installed my brand new hosts HP DL380G8.

Having this in two separate locations. Running iSCSI and two different EMC storages.

Also got the value: 18446744073709552.00

I'm running the following versions:

vCenter 5.5 Update 2b (2183111)

ESXi 5.5 Update 2 (HP-Image)  (2302651)

iSCSI - jumboframes (MTU9000)

This is clearly a bug in the interface, but annoying as I dont get the real number and I'm currently troubleshooting high intermittent latency in my storage and this was throwing VMware support of the topic as we got these numbers. And they told me to check the storage side.

But I finally found this page so I'm not alone with the issue it seems.

I'm sending this to my VMware TAM contact to see if he can get it addressed.

Markus

[kkemp]

This is purely a display issue in esxtop and should only be seen on idle devices.

Once IO is issued and a valid latency value is found it should update and display properly.

There should be a public facing kb detailing the issue soon.

[rtarantola]

Thanks for the valuable info but all of us experiencing this issue I think are still waiting for a patch or new build to resolve this issue.

[jdelvalle]

Also having the same issue and just like another person that posted I found the thread with the number "18446744073709552.00" but there is also another thread I found using the number "9223372036854776.00" which are the 2 numbers I see pop up when running esxtop and then heading viewing the lun (u)stats. Here is my setup:

Hosts

3 HP DL360p Gen 8

SAN

3 Node HP p4300 G2

Networking

2 Cisco 3560 X

[dsohayda]

I am seeing this on an HP Proliant BL685c G7 connected to an EMC VNX7500 via FC.