Weathervane Exceptions

[jiangjing123455]

Hi,

When I started the vmmark test with 4 tiles,the result is "Run_Is_NOT_Compliant".

Warnings Messages::

  p0 : WeathervaneAuction0 Exceptions : 2

  p0 : WeathervaneElastic0 Exceptions : 1

  p1 : WeathervaneAuction0 Exceptions : 4

  p2 : WeathervaneAuction0 Exceptions : 1

  p2 : WeathervaneElastic0 Exceptions : 119

  rampdown : WeathervaneAuction0 Exceptions : 1

  rampdown : WeathervaneElastic0 Exceptions : 120

  p0 : WeathervaneElastic1 Exceptions : 2

  p1 : WeathervaneAuction1 Exceptions : 1

  rampdown : WeathervaneElastic1 Exceptions : 95

  p0 : WeathervaneAuction2 Exceptions : 1

  p0 : WeathervaneElastic2 Exceptions : 1

  p1 : WeathervaneElastic2 Exceptions : 143

  p2 : WeathervaneAuction2 Exceptions : 1

  p2 : WeathervaneElastic2 Exceptions : 173

  rampdown : WeathervaneElastic2 Exceptions : 130

  p0 : WeathervaneAuction3 Exceptions : 3

  p0 : WeathervaneElastic3 Exceptions : 2

  p1 : WeathervaneElastic3 Exceptions : 155

  p2 : WeathervaneElastic3 Exceptions : 183

  rampdown : WeathervaneAuction3 Exceptions : 4

  rampdown : WeathervaneElastic3 Exceptions : 128

Summary ::

Run_Is_NOT_Compliant

Turbo_Setting : 0

Number_of_Workloads_Missing : 0

Number_of_Compliance_Issues (identified by '*' or '+') : 1

Issues Found : 

    Tile3-weathervaneelastic-p0

Median_Phase : p2

 I have two VMware clusters for the SUT and the client systems and an SAN storage system with NVMEs providing the shared iSCSI datastores for infrastructure operations. All systems are connected via  10Gbps ethernet switches.

I looked in the wrf files from weathervane, and I see the following sampling of exceptions:

13:06:14.481 [pool-3-thread-36] WARN c.v.w.w.common.core.Operation - Operation:run restarting userId = 786, operation = GetNextBid, behavior UUID ac322b20-5065-4b90-a786-8277c875b419 Failure Reason = io.netty.handler.timeout.ReadTimeoutException
13:06:15.586 [pool-3-thread-92] WARN c.v.w.w.common.core.Operation - Operation:run restarting userId = 278, operation = GetNextBid, behavior UUID 25cbb910-0112-4722-9d49-fdc1ae1f8c6d Failure Reason = io.netty.handler.timeout.ReadTimeoutException

11:50:42.764 [pool-3-thread-15] WARN c.v.w.w.common.core.Operation - Operation:run Execution Failed for GetNextBid for behavior UUID 98cc3664-6c6a-4dec-9f2e-64f426f4d9d8 Failure Reason = com.vmware.weathervane.workloadDriver.common.exceptions.OperationFailedException: Incomplete response received when retrieving current bid for auction 30171
11:50:42.766 [pool-3-thread-15] WARN c.v.w.w.common.core.Operation - Operation:run restarting userId = 2371, operation = GetNextBid, behavior UUID 98cc3664-6c6a-4dec-9f2e-64f426f4d9d8 Failure Reason = com.vmware.weathervane.workloadDriver.common.exceptions.OperationFailedException: Incomplete response received when retrieving current bid for auction 30171

13:56:21.947 [pool-3-thread-58] WARN c.v.w.w.common.core.Operation - Operation:run Execution Failed for GetNextBid for behavior UUID 71f3adb0-3690-438c-b737-3363419eba37 Failure Reason = java.lang.NullPointerException
13:56:21.947 [pool-3-thread-58] WARN c.v.w.w.common.core.Operation - Operation:run restarting userId = 4626, operation = GetNextBid, behavior UUID 71f3adb0-3690-438c-b737-3363419eba37 Failure Reason = java.lang.NullPointerException

Can you please advise what else could be  causing these issues? Thanks.

[dmorse]

Hi @jiangjing123455 , I am glad to see you were able to scale up to this level.

• 1 compliance issue is very low; have you tried more than one run?  It's quite possible that rerunning may produce a compliant run.
• You mentioned you have SAN storage for the workload VMs; is it all-flash?
• You may want to consider grouping the VMs onto separate LUNs, since some are more I/O intensive, and others are more latency sensitive.

It sounds like the benchmark is running correctly, so now it is just a matter of performance tuning your environment.  Reviewing the published VMmark results (https://www.vmware.com/products/vmmark/results3x.0.html), specifically the "Storage Notes" section of the PDFs that use SAN and vSAN storage, should help here.

[jiangjing123455]

Hi,dmorse

I started the vmmark test with 5 tiles by taking your suggestion，but I only passed the test with 5 tiles for first run.

When I started the wmmark test with 5 tiles for the first time,the result was "Run_Is_Compliant" .

Warnings Messages::

  p0 : WeathervaneAuction0 Exceptions : 4

  p0 : WeathervaneElastic0 Exceptions : 189

  p1 : WeathervaneAuction0 Exceptions : 1

  p1 : WeathervaneElastic0 Exceptions : 345

  p2 : WeathervaneElastic0 Exceptions : 365

  rampdown : WeathervaneElastic0 Exceptions : 323

  p0 : WeathervaneAuction1 Exceptions : 2

  p0 : WeathervaneElastic1 Exceptions : 103

  p1 : WeathervaneAuction1 Exceptions : 1

  p1 : WeathervaneElastic1 Exceptions : 280

  p2 : WeathervaneAuction1 Exceptions : 1

  p2 : WeathervaneElastic1 Exceptions : 602

  rampdown : WeathervaneElastic1 Exceptions : 368

  p0 : WeathervaneAuction2 Exceptions : 1

  p0 : WeathervaneElastic2 Exceptions : 82

  p1 : WeathervaneElastic2 Exceptions : 214

  p2 : WeathervaneElastic2 Exceptions : 290

  rampdown : WeathervaneElastic2 Exceptions : 336

  p0 : WeathervaneElastic3 Exceptions : 117

  p1 : WeathervaneAuction3 Exceptions : 2

  p1 : WeathervaneElastic3 Exceptions : 185

  p2 : WeathervaneAuction3 Exceptions : 1

  p2 : WeathervaneElastic3 Exceptions : 241

  rampdown : WeathervaneElastic3 Exceptions : 386

  p0 : WeathervaneAuction4 Exceptions : 9

  p0 : WeathervaneElastic4 Exceptions : 65

  p1 : WeathervaneAuction4 Exceptions : 1

  p1 : WeathervaneElastic4 Exceptions : 195

  p2 : WeathervaneElastic4 Exceptions : 170

  rampdown : WeathervaneElastic4 Exceptions : 160

Summary ::

Run_Is_Compliant

Median_Phase : p2

But when I started the wmmark test with 5 tiles for the second time,the result was "Run_Is_Not_Compliant" and the number of  'warnings_messages' became more and more.

Warnings Messages::

  p0 : WeathervaneAuction0 Exceptions : 1

  p0 : WeathervaneElastic0 Exceptions : 268

  p1 : WeathervaneElastic0 Exceptions : 345

  p2 : WeathervaneElastic0 Exceptions : 333

  rampdown : WeathervaneElastic0 Exceptions : 381

  p0 : WeathervaneAuction1 Exceptions : 2

  p0 : WeathervaneElastic1 Exceptions : 102

  p1 : WeathervaneElastic1 Exceptions : 261

  p2 : WeathervaneAuction1 Exceptions : 1

  p2 : WeathervaneElastic1 Exceptions : 513

  rampdown : WeathervaneElastic1 Exceptions : 357

  p0 : WeathervaneAuction2 Exceptions : 1

  p0 : WeathervaneElastic2 Exceptions : 20

  p1 : WeathervaneAuction2 Exceptions : 1

  p1 : WeathervaneElastic2 Exceptions : 215

  p2 : WeathervaneAuction2 Exceptions : 1

  p2 : WeathervaneElastic2 Exceptions : 246

  rampdown : WeathervaneElastic2 Exceptions : 282

  p0 : WeathervaneAuction3 Exceptions : 2

  p0 : WeathervaneElastic3 Exceptions : 68

  p1 : WeathervaneElastic3 Exceptions : 211

  p2 : WeathervaneElastic3 Exceptions : 314

  rampdown : WeathervaneElastic3 Exceptions : 327

  p0 : WeathervaneAuction4 Exceptions : 31033

  p0 : WeathervaneElastic4 Exceptions : 70

  p1 : WeathervaneAuction4 Exceptions : 29799

  p1 : WeathervaneElastic4 Exceptions : 212

  p2 : WeathervaneAuction4 Exceptions : 29433

  p2 : WeathervaneElastic4 Exceptions : 292

  rampdown : WeathervaneAuction4 Exceptions : 19909

  rampdown : WeathervaneElastic4 Exceptions : 384

Summary ::

Run_Is_NOT_Compliant

Turbo_Setting : 0

Number_of_Workloads_Missing : 0

Number_of_Compliance_Issues (identified by '*' or '+') : 2

Issues Found : 

    Tile2-weathervaneelastic-p0

    Tile4-weathervaneelastic-p0

Median_Phase : p2

This situation also appeared in the third test and the fourth test.

Can you please advise what else could be  causing these issues? Thanks.

[dmorse · Accepted solution]

Hi @jiangjing123455 , I am not sure why only the first test passes.

If you shut down all 5 tiles using this command from the prime client (make sure you're in the /root/VMmark3 directory first):

java -jar tools/VMmark3Service.jar -c VMmark3.properties -m tilePower -tiles 0

Then start all 5 tiles using this command:
java -jar tools/VMmark3Service.jar -c VMmark3.properties -m tilePower -tiles 5

And then try another 5-tile run, does that pass?