Total newb here, I'm fairly certain I've missed something simple but I cannot seem to figure it out. I have Tile0 that works and will generate a test and results. I provisioned 2 additional tiles, 1 and 2. I have the run section of the properties file modified to list all 3 variants of each of the VMs and tile count set to 3. When I run a new job I see in the job monitor that it is referencing all 3 of the VMs for each type during initialization but once the test kicks off I only see activity on the Tile0 VMs and not any of the 1 or 2 versions. What did I miss?
I attached the properties file I'm using.
Thanks for your time!
-Jason
Something is definitely wrong with the tile1 and tile2 VMs, particularly the client1 and client2 VMs.
If you look at the configfiles/Client[0,1,2]_hosts.txt, you'll see that they all point to tile0 SUT VMs. This is why I see errors like this in the STAX_Job_6_User.log:
20210630-12:23:51 Info Weathervane Auction Abort : Tile 1 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:23:51 Info DS3WebA Abort : Tile 1 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:23:54 Info DS3WebB Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:23:59 Info DS3WebC Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:24:51 Info Weathervane Auction Abort : Tile 2 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:24:52 Info DS3WebA Abort : Tile 2 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:24:55 Info DS3WebB Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:24:58 Info DS3WebC Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
I suggest you delete all tile1 and tile2 VMs (both SUTs and clients), and re-run the provisioning to re-create them. Hopefully then client1 and client2 will have valid /etc/hosts files that point to tile1 and tile2 instead of tile0.
Hi Jason,
Your properties file looks okay to me, so I'll need more information. Could you attach a .zip file of your most recent Results folder? Here's how:
From your primeclient VM:
Then attach the .zip file to your reply
Thanks, David
Thanks @dmorse attached results zip.
Thanks Jason.
Your vCServerVMmark3Cluster and vCServerClientCluster variables are set to the same cluster; the benchmark expects two separate clusters (with different ESXi hosts) for the clients and the SUTs (systems under test).
Another issue: you have SVmotion/SVMotionLUNs = SEA1-VVOL-TEST-02, but you also need to fill in values for XVmotion/XVMotionLUNs, Deploy/DeployLUNs, Deploy/Templates, and Deploy/DeployVMinfo. Here are proposed values, modify to suit your environment:
SVmotion/SVMotionLUNs = SEA1-VVOL-TEST-02
XVmotion/XVMotionLUNs = SEA1-VVOL-TEST-02
Deploy/DeployLUNs = SEA1-VVOL-TEST-02
Deploy/Templates = vmmark3.1.1-template-031420
Deploy/DeployVMinfo = DeployVM0:192.168.1.245,DeployVM1:192.168.1.246
Also, can you give me the exact command you're using to run the benchmark? I'm surprised there weren't more files in your Results directory (I'd expect to see STAX_Job_*.log files for example).
Thanks,
David
I have been using the GUI on the last few attempts by submitting a new job, that last round may have been kicked off by a modified version of the 'VMmark3-STAX-console.sh' script with the config file modified to point to my custom properties file. I'm sorry but I don't recall which method was used for that test but I can re-run in any manner you would like and send the results back.
Additionally, I have the xvmotion and deploy tests unselected in the properties file as well as those options being blank as I do not yet have an environment to run a fully compliant benchmark run. To be fully transparent I only have the 1 cluster to work with currently due to a networking limitation so I do have everything sitting in there. At this stage I am trying to learn how to get the tests to run for when I can get the new network in place for testing across multiple clusters. My end-goal is to use this platform for not only performance checks but as a burn-in tool for new hardware, firmware and ESXi configurations.
I will add your suggested values to the properties file and add those tests back to the list as well, do you think that is why Tile1&2 don't run? Also, let me know if you want me to run the test in a specific manner and re-send the results.
No, Tile1 and Tile2 should have run with that properties file. I would need the entire contents of the Results folder (which include STAX*.log and possibly other files as well) to investigate further, so please do another run and send along a new .zip file.
Also: to greatly shorten the run time of the tests (at least until you have both clusters, and all of the workloads enabled), you could uncomment the TurboMode line and set it to True.
So I tried the command line version first and I think this is probably why you didn't see what you were expecting before. I will re-run via GUI and see if it will produce the expected results data.
Response
--------
{
Job ID : 1
Start Date-Time: 20210630-09:28:26
End Date-Time : 20210630-10:33:38
Status : Terminated
Result : None
Job Log Errors : [
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.
===== XML Information =====
File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 819: Error in element type "message".
===== Python Error Information =====
com.ibm.staf.service.stax.STAXPythonEvaluationException:
Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)
Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined
===== Call Stack for STAX Thread 488 =====
[
sequence: 6/6 (Line: 743, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 768, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 773, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.
===== XML Information =====
File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 819: Error in element type "message".
===== Python Error Information =====
com.ibm.staf.service.stax.STAXPythonEvaluationException:
Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)
Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined
===== Call Stack for STAX Thread 487 =====
[
sequence: 6/6 (Line: 743, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 768, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 773, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.
===== XML Information =====
File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 819: Error in element type "message".
===== Python Error Information =====
com.ibm.staf.service.stax.STAXPythonEvaluationException:
Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)
Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined
===== Call Stack for STAX Thread 490 =====
[
sequence: 6/6 (Line: 743, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 768, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 773, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.
===== XML Information =====
File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 720: Error in element type "message".
===== Python Error Information =====
com.ibm.staf.service.stax.STAXPythonEvaluationException:
Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)
Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined
===== Call Stack for STAX Thread 491 =====
[
sequence: 6/6 (Line: 644, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 669, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 674, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.
===== XML Information =====
File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 720: Error in element type "message".
===== Python Error Information =====
com.ibm.staf.service.stax.STAXPythonEvaluationException:
Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)
Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined
===== Call Stack for STAX Thread 486 =====
[
sequence: 6/6 (Line: 644, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 669, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 674, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
{
Date-Time: 20210630-10:33:38
Level : Error
Message : STAXPythonEvaluationError signal raised. Terminating job.
===== XML Information =====
File: /root/VMmark3/xml/generic_functions.xml, Machine: local
Line 720: Error in element type "message".
===== Python Error Information =====
com.ibm.staf.service.stax.STAXPythonEvaluationException:
Python string evaluation failed for:
'Reporter Results : %s :: %s : Errors %s : Warnings %s' % (VirtServer,PassFailString,Errors,Warnings)
Traceback (most recent call last):
File "<pyEval string>", line 1, in <module>
NameError: name 'PassFailString' is not defined
===== Call Stack for STAX Thread 489 =====
[
sequence: 6/6 (Line: 644, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
if: RC != 0 (Line: 669, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
sequence: 2/2 (Line: 674, File: /root/VMmark3/xml/generic_functions.xml, Machine: local)
]
}
]
Testcase Totals: {
Tests : 5
Passes: 4
Fails : 1
}
}
tail: cannot open ‘/root/VMmark3/results/Results_20210630092826_ForumTestTurbo/VMmark3-STAX_*.log’ for reading: No such file or directory
See /root/VMmark3/results/Results_20210630092826_ForumTestTurbo for details
VMmark3 Benchmark Completed
Hi Jason, that error is definitely unexpected but could be a result of the fact that you're using the same cluster for your SUT and clients.
In any case, you can try to work around this error by uncommenting this line:
// Reporter = true
And setting it to false:
Reporter = false
Hopefully then the benchmark can make further progress.
Something is definitely wrong with the tile1 and tile2 VMs, particularly the client1 and client2 VMs.
If you look at the configfiles/Client[0,1,2]_hosts.txt, you'll see that they all point to tile0 SUT VMs. This is why I see errors like this in the STAX_Job_6_User.log:
20210630-12:23:51 Info Weathervane Auction Abort : Tile 1 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:23:51 Info DS3WebA Abort : Tile 1 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:23:54 Info DS3WebB Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:23:59 Info DS3WebC Abort : Tile 1 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:24:51 Info Weathervane Auction Abort : Tile 2 : Weathervane Auctio
n Terminated Unexpectedly
20210630-12:24:52 Info DS3WebA Abort : Tile 2 : ds3webdriver Process Terminate
d Unexpectedly
20210630-12:24:55 Info DS3WebB Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
20210630-12:24:58 Info DS3WebC Abort : Tile 2 : Iteration 0 ds3webdriver Proce
ss Terminated Unexpectedly
I suggest you delete all tile1 and tile2 VMs (both SUTs and clients), and re-run the provisioning to re-create them. Hopefully then client1 and client2 will have valid /etc/hosts files that point to tile1 and tile2 instead of tile0.
Thanks. I will strip out the tiles and re-run provisioning. Hopefully I will be able to report back with better results!
Success! I got a good run on all 3 tiles after deleting and redeploying 1 & 2. Thanks for your help!