levans01's Posts

1.0 Background: [Setup] I have a problem I hope someone here can help me with.  I have a large vSphere 6.0 with a few hundred hosts. These hosts are distributed between 4-5 vCenter environments. I have several HPE Nimble storage arrays organized into groups and they present various LUNs to the various vCenter environments. The vCenter environments are organized into two groups. A couple are for Hosting servers as guests and the others are for VDI. The storage arrays are arranged in a similar fashion so that one array cluster only presents LUNs to vCenters that have servers as guests and the other array presents LUNs to the vCenters used for VDI. The exception to this are two volumes.  We have  one volume that is used to present template and ISO images located in the SAN used to present LUNs to the VDI environment that is also made available to the vCenters for Servers.  And another volume that is hosted on the SAN for Servers and is presented to all  environments for temporary storage and the movement of data between the various vCenters.  This LUN is generally empty and is not used for heart-beating etc. All volumes are mounted using ISCSI software Adapter. 2.0  The Problem: In troubleshooting an unrelated issue with the storage vendor we noted these lightly used volumes were showing up really high on the IOPS and Network throughput list higher than many of our production LUNs some of which contain 200-300 VMs. IOPS and and throughput for these volumes were way out of line for their use.  We were able to determine this IO activity was not related to any actual workloads so it must be coming from the hosts or is VMware/ESXi related in some way. 3.0 What has been Done: We opened a ticket with VMware support.  They ran through all of the storage related performance checks and did not find anything out of order.  We disabled ATS on the hosts to see if it would change anything it did not.  We were able to determine the scope to be as follows of the 5 vCenters the two volumes are mounted to ,the problem only shows up on three of the 5 vCenters  1 vCenter for servers and 2 VDI vCenters. Our largest vCenter which contains 4 clusters the problem only shows up in two of the clusters. We know that if we unmount the volume the problem appears to go away.  We see corresponding drops in IOPs and throughput. We’ve taken packet traces and checked the ESXI hosts logs and we found something interesting but have not been able to get anything to line up.  We can see the high-IOPS/throughput when  Setting a performance chart  Performance > Datastore > Realtime > (datastores) > Read Rate.   In places where this problem does not exists, the throughput is  < 20 KBps  but in places were we are seeing the problem we see 3000-8000KBps.  Most of the errors we are seeing in the vmkernel.log on the hosts are MISCOMPARE ON VERIFY Please let me know if additional information is needed. Thank you.

Issue Details:  1. We use PowerCLI to get values from VMs.  There are instances where the values fail to return.  It is our belief that this worked flawlessly before and we can show that it still does in some instances.  We developed a test script to reproduce the problem as well as show instances where things appear to work correctly.  This script connects to a machine we chose.  We provide administrative credentials and the script returns the value 12345.  It does ten iterations of this task.  We expect 10/10.  $creds = Get-Credential $successes = 0; while ($true) { $output = Invoke-VMScript -VM "MyVM" -ScriptText "return '12345'" -GuestCredential $creds $output if ($output.ScriptOutput) {$successes++} if ($successes -eq 10) {break} } When we run this script against some servers it works flawlessly.  That is the return is 10/10.  When we look at the VM > Tasks & Events > Events  we see the following successive events which are normal. (1) Guest operation Create Temporary File performed. (2) Guest operation Start program performed. (3) Guest operation List processes Performed. (4) Guest operation List processes Performed. (5) Guest operation Initiate File Transfer from Guest Perform. (6) Guest operations Delete File performed. The system repeats these events for each successful iteration. 2.  Our test script will make as many iterations as possible to get 10 successful results.   Ideally, it would only have to make 10 attempts to get 10 results.  But when there are failures, we have to make more than 10 attempts to get the desired results. I have seen it take as many as 19 attempts to get 10 successes. 3. As an additional data point, When running this script against the domain controller in an environment it works flawlessly.  However, when running against a domain joined machine in the domain, the problem happens. If I watch the creation of the Powershell## file in the temp directory on the system.  The ones that copy with 0 bytes are the ones that fail.  Any ideas? Thanks!