VMware Performance Community
Foyman1973
Enthusiast
Enthusiast
Jump to solution

Activity only on Tile0 VMmark 3.1.1

Total newb here, I'm fairly certain I've missed something simple but I cannot seem to figure it out.  I have Tile0 that works and will generate a test and results.  I provisioned 2 additional tiles, 1 and 2.  I have the run section of the properties file modified to list all 3 variants of each of the VMs  and tile count set to 3.  When I run a new job I see in the job monitor that it is referencing all 3 of the VMs for each type during initialization but once the test kicks off I only see activity on the Tile0 VMs and not any of the 1 or 2 versions.  What did I miss? 

I attached the properties file I'm using.

Thanks for your time!

-Jason

Labels (1)
Reply
0 Kudos
1 Solution

Accepted Solutions
dmorse
VMware Employee
VMware Employee
Jump to solution

Something is definitely wrong with the tile1 and tile2 VMs, particularly the client1 and client2 VMs.

If you look at the configfiles/Client[0,1,2]_hosts.txt, you'll see that they all point to tile0 SUT VMs.  This is why I see errors like this in the STAX_Job_6_User.log:

20210630-12:23:51 Info Weathervane Auction Abort : Tile 1 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:23:51 Info DS3WebA Abort : Tile 1 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:23:54 Info DS3WebB Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:23:59 Info DS3WebC Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly

20210630-12:24:51 Info Weathervane Auction Abort : Tile 2 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:24:52 Info DS3WebA Abort : Tile 2 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:24:55 Info DS3WebB Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:24:58 Info DS3WebC Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly

I suggest you delete all tile1 and tile2 VMs (both SUTs and clients), and re-run the provisioning to re-create them.  Hopefully then client1 and client2 will have valid /etc/hosts files that point to tile1 and tile2 instead of tile0.

View solution in original post

11 Replies
dmorse
VMware Employee
VMware Employee
Jump to solution

Hi Jason,

Your properties file looks okay to me, so I'll need more information.  Could you attach a .zip file of your most recent Results folder?  Here's how:

From your primeclient VM:

  1. Login as root
  2. cd /root/VMmark3/results
  3. zip your results folder, i.e. "zip -r result.zip Results_20210625092906"

Then attach the .zip file to your reply

Thanks, David

Reply
0 Kudos
Foyman1973
Enthusiast
Enthusiast
Jump to solution

Thanks @dmorse attached results zip.

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Thanks Jason.

Your vCServerVMmark3Cluster and vCServerClientCluster variables are set to the same cluster; the benchmark expects two separate clusters (with different ESXi hosts) for the clients and the SUTs (systems under test).

Another issue: you have SVmotion/SVMotionLUNs = SEA1-VVOL-TEST-02, but you also need to fill in values for XVmotion/XVMotionLUNs, Deploy/DeployLUNs, Deploy/Templates, and Deploy/DeployVMinfo.  Here are proposed values, modify to suit your environment:

SVmotion/SVMotionLUNs = SEA1-VVOL-TEST-02
XVmotion/XVMotionLUNs = SEA1-VVOL-TEST-02
Deploy/DeployLUNs = SEA1-VVOL-TEST-02
Deploy/Templates = vmmark3.1.1-template-031420
Deploy/DeployVMinfo = DeployVM0:192.168.1.245,DeployVM1:192.168.1.246

Also, can you give me the exact command you're using to run the benchmark?  I'm surprised there weren't more files in your Results directory (I'd expect to see STAX_Job_*.log files for example).

Thanks,
David

Reply
0 Kudos
Foyman1973
Enthusiast
Enthusiast
Jump to solution

I have been using the GUI on the last few attempts by submitting a new job, that last round may have been kicked off by a modified version of the 'VMmark3-STAX-console.sh' script with the config file modified to point to my custom properties file.  I'm sorry but I don't recall which method was used for that test but I can re-run in any manner you would like and send the results back.  

Additionally, I have the xvmotion and deploy tests unselected in the properties file as well as those options being blank as I do not yet have an environment to run a fully compliant benchmark run.  To be fully transparent I only have the 1 cluster to work with currently due to a networking limitation so I do have everything sitting in there.  At this stage I am trying to learn how to get the tests to run for when I can get the new network in place for testing across multiple clusters.  My end-goal is to use this platform for not only performance checks but as a burn-in tool for new hardware, firmware and ESXi configurations.

I will add your suggested values to the properties file and add those tests back to the list as well, do you think that is why Tile1&2 don't run?  Also, let me know if you want me to run the test in a specific manner and re-send the results.

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

No, Tile1 and Tile2 should have run with that properties file.  I would need the entire contents of the Results folder (which include STAX*.log and possibly other files as well) to investigate further, so please do another run and send along a new .zip file.

Also: to greatly shorten the run time of the tests (at least until you have both clusters, and all of the workloads enabled), you could uncomment the TurboMode line and set it to True.

Reply
0 Kudos
Foyman1973
Enthusiast
Enthusiast
Jump to solution

So I tried the command line version first and I think this is probably why you didn't see what you were expecting before.  I will re-run via GUI and see if it will produce the expected results data.

 

Response
--------
{
Job ID : 1
Start Date-Time: 20210630-09:28:26
End Date-Time : 20210630-10:33:38
Status : Terminated
Result : None
Job Log Errors : [
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.

===== XML Information =====

File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 819: Error in element type "message".

===== Python Error Information =====

com.ibm.staf.service.stax.STAXPythonEvaluationException:

Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)

Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined

===== Call Stack for STAX Thread 488 =====

[
sequence: 6/6 (Line: 743, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 768, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 773, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.

===== XML Information =====

File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 819: Error in element type "message".

===== Python Error Information =====

com.ibm.staf.service.stax.STAXPythonEvaluationException:

Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)

Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined

===== Call Stack for STAX Thread 487 =====

[
sequence: 6/6 (Line: 743, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 768, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 773, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.

===== XML Information =====

File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 819: Error in element type "message".

===== Python Error Information =====

com.ibm.staf.service.stax.STAXPythonEvaluationException:

Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)

Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined

===== Call Stack for STAX Thread 490 =====

[
sequence: 6/6 (Line: 743, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 768, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 773, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.

===== XML Information =====

File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 720: Error in element type "message".

===== Python Error Information =====

com.ibm.staf.service.stax.STAXPythonEvaluationException:

Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)

Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined

===== Call Stack for STAX Thread 491 =====

[
sequence: 6/6 (Line: 644, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 669, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 674, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.

===== XML Information =====

File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 720: Error in element type "message".

===== Python Error Information =====

com.ibm.staf.service.stax.STAXPythonEvaluationException:

Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)

Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined

===== Call Stack for STAX Thread 486 =====

[
sequence: 6/6 (Line: 644, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 669, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 674, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.

===== XML Information =====

File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 720: Error in element type "message".

===== Python Error Information =====

com.ibm.staf.service.stax.STAXPythonEvaluationException:

Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)

Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined

===== Call Stack for STAX Thread 489 =====

[
sequence: 6/6 (Line: 644, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 669, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 674, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
]
Testcase Totals: {
Tests : 5
Passes: 4
Fails : 1
}
}
tail: cannot open ‘/root/VMmark3/results/Results_20210630092826_ForumTestTurbo/VMmark3-STAX_*.log’ for reading: No such file or directory

See /root/VMmark3/results/Results_20210630092826_ForumTestTurbo for details

VMmark3 Benchmark Completed

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Hi Jason, that error is definitely unexpected but could be a result of the fact that you're using the same cluster for your SUT and clients.

In any case, you can try to work around this error by uncommenting this line:
// Reporter = true

And setting it to false:
Reporter = false

Hopefully then the benchmark can make further progress.

Reply
0 Kudos
Foyman1973
Enthusiast
Enthusiast
Jump to solution

Thanks, disabling the reporter seemed to create a full set of files this time, and solved the mystery of why I stopped getting email notices when jobs completed!  I have attached a new results bundle, hopefully it's all there this time.

-Jason

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Something is definitely wrong with the tile1 and tile2 VMs, particularly the client1 and client2 VMs.

If you look at the configfiles/Client[0,1,2]_hosts.txt, you'll see that they all point to tile0 SUT VMs.  This is why I see errors like this in the STAX_Job_6_User.log:

20210630-12:23:51 Info Weathervane Auction Abort : Tile 1 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:23:51 Info DS3WebA Abort : Tile 1 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:23:54 Info DS3WebB Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:23:59 Info DS3WebC Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly

20210630-12:24:51 Info Weathervane Auction Abort : Tile 2 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:24:52 Info DS3WebA Abort : Tile 2 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:24:55 Info DS3WebB Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:24:58 Info DS3WebC Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly

I suggest you delete all tile1 and tile2 VMs (both SUTs and clients), and re-run the provisioning to re-create them.  Hopefully then client1 and client2 will have valid /etc/hosts files that point to tile1 and tile2 instead of tile0.

Foyman1973
Enthusiast
Enthusiast
Jump to solution

Thanks.  I will strip out the tiles and re-run provisioning.  Hopefully I will be able to report back with better results!

Reply
0 Kudos
Foyman1973
Enthusiast
Enthusiast
Jump to solution

Success!  I got a good run on all 3 tiles after deleting and redeploying 1 & 2.  Thanks for your help!