The DAVG, from what I understand, is considered normal under 20-25. Generally I see anywhere from 0.x to about 20. Then when this problem is occurring it goes into the 10's of thousands for that particular LUN for 10-20 seconds. I have been able to reproduce this problem by copying a large (3gb) file to RDM's attached to some VM's. I can see the file copy freeze, the DAVG's skyrocket, the VM become unresponsive. Once the VM becomes responsive, the file copy finishes and then I see the LSA_SAS Device Reset event show up.
In your array are your hosts set for Active/active ALUA or active/passive? Are you running Powerpath VE on your hosts? or are you using NMP set to round robin?
I ask this, because I found something very interesting today. From one host I uninstalled PowerPath and ran my test using NMP Round Robin and saw the same exact results. In storage adapters, on that particular LUN I changed it to Fixed Path, and ran the test on each vmhba (setting each to preferred) one at a time, performing the same file copy several times. And not once did this problem occur! It seems that it is a multipathing problem. On my array my hosts are set to Active/Active, so I'm not sure yet if I change that to Active Passive and run it in Round Robin if it will work properly. I'm testing that next. I talked to the VMware engineer that I have been working this problem with, and he said it may be a problem on the array side. I also need to check compatibility for vsphere 5.1 update 2, Active/Active Alua, and UCS 2.2(1d) to make sure it should all work together. Maybe upgrading to vsphere 5.5 will fix it? that may be my next step, before considering downgrading UCS firmware. I haven't had my environment running on anything other than 5.1U2 and 2.2(1d), as I migrated from older Dell physical hosts running vsphere 4.1directly to the B200m3 hosts on 5.1u2.
