VMware Performance Community
Pavel2337
Contributor
Contributor
Jump to solution

VMmark3 test runs render 0 VMmark3 and Infrastructure Scores

My VMmark3 deployment has started producing 0 scores for no obvious reasons. No changes has been done to environment except VCenter version was upgraded to Version 14367737

VMmark3 infrastructure has been cleaned and redeployed including PrimeClient and Tile0. Also I have noticed that some Deploy, sVmotion and xVmotion operations didn't get completed and just timed out during VMmark3 test run even though they are done on VCenter. There is no connectivity problem between PrimeClient and VCenter.

Any help or suggestions would be greatly appreciated.

Reply
0 Kudos
1 Solution

Accepted Solutions
RebeccaG
Expert
Expert
Jump to solution

To sum up, the reason you were getting 0 score is a load-related issue where the Deploy wasn't finishing.

If you compare the 6-tile result and the 8-tile result, you can see your SVMotion, XVMotion and Deploy latencies have doubled. This is where to look in Score_8_Tile_Test_NC.txt:

pastedImage_0.png

That would implicate the storage, or I suppose the vCenter Server is also involved in the operation. I'm assuming the environment was the same between both runs.

The timestamp on both the 6-tile and 8-tile runs is similar, April 13. Did you run the 8-tile run first and then the 6 tile run second? All the QoS looks really good on the 6-tile run, and terrible on the 8-tile run.

View solution in original post

Reply
0 Kudos
13 Replies
RebeccaG
Expert
Expert
Jump to solution

Hi Pavel,

I'd like to see the logs from the first run on which you saw this problem, and the most recent run, since you said you redeployed. Please zip the results folders and attach them to this post. If it's too large, message me and I will show you an alternate way to upload the file.

That build of vCenter has been broadly used with VMmark, so it's not likely that is a problem itself.

Am I correct in understanding that after you first saw the issue of 0 score, you redeployed the PrimeClient and Tile0 and continued to see the same issue?

If you are redeploying from your original VMmark ova, please make sure you have applied patch 3.1.1. If you haven't, you can download a new OVA from vmmark.com now that has 3.1.1 installed.

Thank you,

Rebecca

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

Zip file is bigger than 1GB so I need alternative way of uploading. Also I might not find first failed one.

Yes, you are correct - I have redeployed after I saw first 0 score run. And somewhere between first 0 score and redeployment I got VCenter upgraded.

As of OVA - yes, I am using 3.1.1

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

Unfortunately first 0 score run is gone with previous PrimeClient VM being deleted.

Reply
0 Kudos
RebeccaG
Expert
Expert
Jump to solution

OK, I have messaged you about uploading the result. Please check your forum inbox.

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

I'm new to VMware forum. Where can I find my forum inbox?

Reply
0 Kudos
RebeccaG
Expert
Expert
Jump to solution

The direct URL is https://communities.vmware.com/inbox

You should see an "@" symbol like this at the top of the page which links there:

pastedImage_0.png

I had trouble finding the link too, to be honest.

Reply
0 Kudos
RebeccaG
Expert
Expert
Jump to solution

Thank you for sending the results files over. This does appear to be a load related issue.

The reason why the 8-tile run had a score and the 10-tile run had a 0 score is that the 8-tile run had some passing Deploy operations, whereas the 8-tile run had 0 passing operations. When 0 operations (of any workload) pass during a VMmark run, the score is 0. However, both runs still failed Quality of Service overall.

On both the 8-tile run and 10-tile run, there were a large number of Quality of Service failures, although there were fewer failures on the 8-tile run. Most likely, either the storage or the system under test's CPU is overloaded. Try doing a 1-tile run and send me the results if you have a problem.

Were you previously getting passing VMmark runs (i.e. Score_N_Tile_Test.txt has the text "Run_Is_Compliant") on this same environment?

Thank you,

Rebecca

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

Thank you, Rebecca.

Yes, I getting compliant runs of 6 tiles, please see attached score file. Compliance issues is ok for now as we are trying to tune system to get better score. So correct me if I'm wrong here - I'm facing load related issues on storage side as I'm not seeing ESXi hosts running up to 100%.

Reply
0 Kudos
RebeccaG
Expert
Expert
Jump to solution

To sum up, the reason you were getting 0 score is a load-related issue where the Deploy wasn't finishing.

If you compare the 6-tile result and the 8-tile result, you can see your SVMotion, XVMotion and Deploy latencies have doubled. This is where to look in Score_8_Tile_Test_NC.txt:

pastedImage_0.png

That would implicate the storage, or I suppose the vCenter Server is also involved in the operation. I'm assuming the environment was the same between both runs.

The timestamp on both the 6-tile and 8-tile runs is similar, April 13. Did you run the 8-tile run first and then the 6 tile run second? All the QoS looks really good on the 6-tile run, and terrible on the 8-tile run.

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

Thanks again.

Yes, 6 tiles run was done after 8 tiles run. And it's been done on the same environment/deployment, I just removed 2 tiles. Storage slowness might explain those incomplete operations, but is there specific requirements for VCenter? Also is there a way to check/test VCenter?

And it brings question in my original posting. I observed Deploy, sVMotion and xVmotion operations hung on STAX Job Monitor while VCenter has reported them completed. Is there any way to find out why?

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

Here is example of hung Deploy operation:

pastedImage_0.png

As you can see Deploy operation is still running while DeployVM0 already deployed and also running next operation of creating DeployVM0 Snapshot. Unless snapshot creation is part Deploy operation and actually causing high time of deploy operation. Can you clarify on this? 

Reply
0 Kudos
RebeccaG
Expert
Expert
Jump to solution

You might see this behavior also if you've overcommitted memory on your systems under test or client systems. Could you please check that you've met all the requirements in "VMmark Hardware Requirements" in the VMmark User's Guide?

is there specific requirements for VCenter?

Yes, please see the VMmark User's Guide, section "vCenter Hardware Requirements".

Also is there a way to check/test VCenter?

Not specifically, I would recommend at least checking its CPU and memory usage during a VMmark run.

I observed Deploy, sVMotion and xVmotion operations hung on STAX Job Monitor while VCenter has reported them completed. Is there any way to find out why?

You said in a later post:

Unless snapshot creation is part Deploy operation and actually causing high time of deploy operation.

That's correct, the VMmark Deploy operation is a set of operations which includes deploy, snapshotting, etc. When the entire string of events is finished, one VMmark Deploy operation has completed. On your 8-tile run, it was taking roughly one hour to complete these operations. VMmark's baseline 'normal' execution time for Deploy (slightly older systems, tests completed in 2017) was 6 minutes.

Reply
0 Kudos
Pavel2337
Contributor
Contributor
Jump to solution

Thank you, Rebecca. I think I'm good for now.

Reply
0 Kudos