All VMware VMmark™ Discussions posts

Re: VMmark Run Tiles Target and Source Datastorages are the same

RebeccaG — Thu, 02 Nov 2023 21:56:35 GMT

Thank you Oporras for uploading your results files.

These are the VMmark storage infrastructure operations VMs: (quoted from VMmark3.properties file)

// SVmotion/Workloads = Standby

// XVmotion/Workloads = DS3WebA

Please make sure that these VMs for every tile are located on a different datastore from ones you specified in VMmark3.properties:

SVmotion/SVMotionLUNs = datastore1,datastore1

XVmotion/XVMotionLUNs = datastore1,datastore1

You can actually use the same datastore for all infrastructure operations as shown above, but they can also all be different.

The error you are seeing is because the workload VM in question is present on the same datastore as specified in SVMotionLUNs or XVMotionLUNs. It needs to be on a different datastore. This is why VMmark 3 requires at least 2 shared datastores across the SUT cluster.

In file Results_20231029035855_tile0, you have

SVmotion/SVMotionLUNs = DS_CLIPTEST01,DS_CLIPTEST02

You stated:

DS_CLIPTEST01 >
Standby0

Standby0 is on DS_CLIPTEST01, that is the source of the problem. Please migrate Standby0 to a different datastore or specify a different datastore in SVmotion/SVMotionLUNs.

In Results_20231029031758,

SVmotion/SVMotionLUNs = DS_AMSA_QA_DEV,DS_BASESWIN_QA_01

You stated:

DS_AMSA_QA_DEV >
Standby0

Same problem.

Also, I did not do a comprehensive review of your setup but I wanted to mention a couple of things I noticed:

1. Make sure you do not run VMmark on a production environment. There were many other VMs on your VMmark cluster. From the VMmark 3 User's Guide, "NOTE VMmark, and the benchmark hardware and software components (vCenter, ESXi, and the VMmark template, clients, workload virtual machines, systems under test, storage, etc.) must be used only in an isolated test environment. None of the benchmark hardware or software components should ever be connected to a production network or transitioned to a production environment." Among other things, when you run VMmark it will create terrible performance for any production applications and produce invalid VMmark 3 results.

2. You have specified the same cluster for both the VMmark 3 Client cluster and SUT cluster. These need to be separate clusters and physical machines. You can still run VMmark with this configuration, for example, for functional testing, but it will produce invalid performance results.

Thank you,

Rebecca

Re: how many Tiles can I run?

RebeccaG — Tue, 31 Oct 2023 20:17:03 GMT

Hi oporras,

VMmark is typically used to run as many tiles as possible in a cluster and still achieve a passing score.

To find a rough estimate of how many tiles your cluster will support, you can look at the VMmark results page. Try filtering results by the total number of cores you have in your cluster. You may also be able to find a published VMmark result which uses the same or similar processor to your cluster, and extrapolate the number of tiles.

You are also thinking about the memory and storage requirements for VMmark, that is good.

Check the "System Requirements" on the VMmark website https://www.vmware.com/products/vmmark.html as it lists the amount of memory and storage needed for each tile.

After you provision VMmark 3 with the tiles you calculated, see whether the run passed or failed (see the VMmark User's Guide for details).

If it passed, you need to provision more tiles so you can find the largest number of tiles that passes.

If it failed you need to run VMmark 3 with one less tile. Continue until you have a passing run.

Re: VMmark3 tiles do I run for 150VM?

RebeccaG — Tue, 31 Oct 2023 20:16:22 GMT

Hi oporras,

VMmark 3 is typically used to run as many tiles as possible in a cluster and still achieve a passing score.

You are also thinking about the memory and storage requirements for VMmark, that is good.

Check the "System Requirements" on the VMmark website https://www.vmware.com/products/vmmark.html as it lists the amount of memory and storage needed for each tile.

After you provision VMmark 3 with the tiles you calculated, see whether the run passed or failed (see the VMmark User's Guide for details).

If it passed, you need to provision more tiles so you can find the largest number of tiles that passes.

If it failed you need to run VMmark 3 with one less tile. Continue until you have a passing run.

VMmark Run Tiles Target and Source Datastorages are the same

oporras — Sun, 29 Oct 2023 15:48:11 GMT

I'm having trouble completing the "VMmark Run Tiles" phase for VMmark 3.1.1
All this occurs in the phase of executing the tiles. (VMmark Run Tiles Fase)
In the properties file I have tried different definitions:

Case 1 - Results_20231029031758_tile0.zip
Settings:
SVmotion/SVMotionLUNs = datastorage1,datastorage2
XVmotion/XVMotionLUNs = datastorage3,datastorage4
Deploy/DeployLUNs = datastorage5
------------- All different data storage -------------
File configuration:
SVmotion/SVMotionLUNs = DS_AMSA_QA_DEV,DS_BASESWIN_QA_01
XVmotion/XVMotionLUNs = DS_CLIPTEST02,DS_AMSADEV_01
Deploy/DeployLUNs = DS_CLIPTEST01

VM location on Tile0 for each data storage
DS_AMSA_QA_DEV >
Standby0
DS_CLIPTEST02 >
DS3WebA
DS_CLIPTEST01 >
vmmark3.1.1-template-031420 (VM status off)
Rest of the Tiles0 (DS3DB0, AuctionLB0, DS3WebB0, etc.)

Case 2 - Results_20231029035855_tile0.zip
Settings:
SVmotion/SVMotionLUNs = datastorage1,datastorage2
XVmotion/XVMotionLUNs = datastorage2,datastorage1
Deploy/DeployLUNs = datastorage1
--------- Using two different data storages ---------
--------- This is the expected configuration --------
File configuration:
SVmotion/SVMotionLUNs = DS_CLIPTEST01,DS_CLIPTEST02
XVmotion/XVMotionLUNs = DS_CLIPTEST02,DS_CLIPTEST01
Deploy/DeployLUNs = DS_CLIPTEST01

VM location on Tile0 for each data storage
DS_CLIPTEST01 >
Standby0
DS_CLIPTEST02 >
DS3WebA
DS_CLIPTEST01 >
vmmark3.1.1-template-031420 (VM status off)
Rest of the Tiles0 (DS3DB0, AuctionLB0, DS3WebB0, etc.)

The error persists for any of the cases.
The VMmark-STAX_Job_2_User-Parsed log shows at the end of the line that the source data storage is the same as the target

Info: DS3 Tile0 Starting with 100GB DB for 10800 secs
InfraOps: Request sVMotion0 #1 : VM Standby0
InfraOps: Request Deploy0 #0 : DeployVM0 onto SUT Host04.XXX.local in LUN DS_CLIPTEST01 using 11.11.11.11
Service Requested Stop: ExceptionFound: SVMotion TargetDatastore Exception : Review Log and Configuration : :Target and Source Datastores are the same
VMmark3: Service Requested Stop: ExceptionFound: SVMotion TargetDatastore Exception : Review Log and Configuration : :Target and Source Datastores are the same : 90.04 (secs)

Additionally, the VMmark3-RunService log shows an error in an attempt to execute the SVMotion

INFO ST20 : Relocation Storage VMotion VM: DS3WebA0 : HostsList Host01.XXX.local;Host04.XXX.local;Host05.XXX.local;Host03.XXX.local;Host02.XXX.local: SourceHost Host01.XXX.local
INFO ST20 : Relocation Storage VMotion VM HostsList Size: 5
INFO ST20 : Info: Relocation Storage VMotion VM : CurrentHostIndex 0 WantedHostIndex 0 AltHostIndex 1)
INFO ST20 : RelocateStorageVM VMotion Unecessary
INFO ST20 : Relocation Storage VMotion VM: Standby0 : HostsList Host01.XXX.local;Host04.XXX.local;Host05.XXX.local;Host03.XXX.local;Host02.XXX.local: SourceHost Host01.XXX.local
INFO ST20 : Relocation Storage VMotion VM HostsList Size: 5
INFO ST20 : Info: Relocation Storage VMotion VM : CurrentHostIndex 0 WantedHostIndex 0 AltHostIndex 1)
INFO ST20 : RelocateStorageVM VMotion Unecessary
ERROR MAIN : Exception Caught: SVMotion TargetDatastore Exception : Review Log and Configuration : Target and Source Datastores are the same : exitSetting true
INFO MAIN : Email Disabled
INFO MAIN : Starting StopFile Creation
INFO MAIN : Ending Early

how many Tiles can I run?

oporras — Thu, 26 Oct 2023 23:42:22 GMT

With 240 cpu, 7.5TB of ram and 16TB of storage, how many Tiles can I run?

thank

Re: VMmark3 tiles do I run for 150VM?

oporras — Thu, 26 Oct 2023 22:38:05 GMT

How do you determine the number of VMs.
With 240 cpu, 7.5TB of ram and 16TB of storage, how many Tiles can I run?

thank

Re: VMmark3 tiles do I run for 150VM?

RebeccaG — Thu, 26 Oct 2023 21:55:33 GMT

Hi, each VMmark3 tile runs 19 VMs on the SUT (system under test) cluster.

7 tiles will run 133 VMs.

8 tiles will run 152 VMs.

Thank you,

Rebecca

VMmark3 tiles do I run for 150VM?

oporras — Thu, 26 Oct 2023 19:52:20 GMT

If I want to run 150VM how many tiles do I run?

Re: two questions in vmmark test

RebeccaG — Tue, 24 Oct 2023 22:48:31 GMT

Hi Wen, this is great. I see you resolved your earlier problems and are running 10 tiles with no setup issues.

Number_of_Compliance_Issues (identified by '*' or '+') : 1 Issues Found : Tile4-weathervaneauction-p1 Median_Phase : p1

This is a Quality of Service (QoS) compliance issue. It means that the VMmark run failed to meet quality of service (i.e. Application Latency) requirements due to resource contention.

This is normal when you are running a large number of tiles as eventually the QoS will fail due to storage contention, CPU contention, or other resource constraints.

The failing tile, tile 4 is just barely failing. A passing score is less than 1.

In VMmark3-graphs.html, it shows WeathervaneAuction has two spikes in QoS during the middle of the run. This is why the QoS failed. If you can resolve the cause of this poor performance, you might be able to pass a VMmark run at 10 tiles or more. You could also run VMmark again, and might be able to pass if you get lucky (slightly better performance from your environment).

Thank you,

Rebecca

Re: two questions in vmmark test

xiaolei_xue — Tue, 24 Oct 2023 06:13:49 GMT

Hi RebeccaG I have a issue in the Score.txt . Number_of_Compliance_Issues (identified by '*' or '+') : 1 Issues Found : Tile4-weathervaneauction-p1 Median_Phase : p1 I have uploaded the zip log on the ftp, the the directory name is 'Wen',and the filenames are Results_20231023072053_10tiles_2.zip.00*. Please use 7-zip to decompress the six packages together. Thank You Wen

Re: VMmark3 Properties

RebeccaG — Mon, 23 Oct 2023 19:07:12 GMT

Hi oporras,

Please don't include spaces or special characters in vSphere names (usernames, datacenters, clusters, etc.). You can see this is in the VMmark User's Guide under "Configure vCenter Server".
Once you have fixed that, if you are still having a problem, please zip the VMmark results folder and upload it as an attachment to this forum thread. This will help me troubleshoot the issue.

Thank you,

Rebecca

VMmark3 Properties

oporras — Sun, 22 Oct 2023 20:04:48 GMT

Good afternoon. I am trying to run VMmark3 in a VMware vsphere environment with 5 clustered hosts. However, when I do the provisioning, I get an error that the Data Center is not recognized. Does someone know how to solve this problem? Thank you in advance for your help.

Re: two questions in vmmark test

RebeccaG — Mon, 16 Oct 2023 15:44:58 GMT

Hi Wen,

It should be ok if those hosts are in a different datacenter with same vCenter Server.

For the new error, please zip your VMmark results file and if small enough, attach to this forum thread. If the zip file is too large please put it on ftpsite like last time.

Also please try to manually vmotion or storage vmotion your VMmark VMs as it is failing in your environment. Also please check why your vMotion LUN shows error from my last post, it is the column 'uncommitted' which shows the error.

Your file VC-ListLUNs.txt also shows an error with the vmotion LUN:

Name : Free Space : Capacity : Uncommitted : Type : Multiple Host Access : Maintenance Mode
VMotion : 7819GB : 7821GB : ERROR : VMFS : true : normal

Second, all infrastructure operations (vmotion, xvmotion, svmotion, deploy) .wrf files show the response "FAIL". This error from vCenter Server means the operation cannot complete. To troubleshoot this, replicate the activity manually:

Using the vCenter UI, manually vmotion, storage vmotion or combined vmotion and storage vmotion a VMmark VM from one host to the other host in the SUT cluster.

Does it complete successfully? If not, watch the error that vCenter Server returns. You will need to fix this before you can run VMmark.

Re: two questions in vmmark test

xiaolei_xue — Fri, 13 Oct 2023 07:25:16 GMT

Hi RebeccaG, Here is another problem as below.Why the test runs too short. 20231013-02:00:50 Info Error: tile 1 DS3WebA run too short, elapsed time = 50,expected time = 189 20231013-02:00:50 Info Error: tile 1 AuctionLB run too short, elapsed time = 50, expected time = 11220 20231013-02:01:50 Info Error: tile 2 AuctionLB run too short, elapsed time = 50, expected time = 11160 20231013-02:01:51 Info Error: tile 2 DS3WebA run too short, elapsed time = 51,expected time = 188 20231013-02:02:50 Info Error: tile 3 DS3WebA run too short, elapsed time = 50,expected time = 187 20231013-02:02:50 Info Error: tile 3 AuctionLB run too short, elapsed time = 50, expected time = 11100 20231013-02:03:50 Info Error: tile 4 DS3WebA run too short, elapsed time = 50,expected time = 186 20231013-02:03:50 Info Error: tile 4 AuctionLB run too short, elapsed time = 50, expected time = 11040 20231013-02:04:50 Info Error: tile 5 AuctionLB run too short, elapsed time = 50, expected time = 10980 20231013-02:04:51 Info Error: tile 5 DS3WebA run too short, elapsed time = 51,expected time = 185 20231013-02:05:50 Info Error: tile 6 AuctionLB run too short, elapsed time = 50, expected time = 10920 20231013-02:05:51 Info Error: tile 6 DS3WebA run too short, elapsed time = 51,expected time = 184 20231013-02:06:50 Info Error: tile 7 DS3WebA run too short, elapsed time = 50,expected time = 183 20231013-02:06:50 Info Error: tile 7 AuctionLB run too short, elapsed time = 50, expected time = 10860 20231013-02:07:50 Info Error: tile 8 AuctionLB run too short, elapsed time = 50, expected time = 10800 20231013-02:07:51 Info Error: tile 8 DS3WebA run too short, elapsed time = 51,expected time = 182 20231013-02:08:50 Info Error: tile 9 DS3WebA run too short, elapsed time = 50,expected time = 181 20231013-02:08:50 Info Error: tile 9 AuctionLB run too short, elapsed time = 50, expected time = 10740 20231013-02:09:50 Info Error: tile 10 AuctionLB run too short, elapsed time =50, expected time = 10680 20231013-02:09:51 Info Error: tile 10 DS3WebA run too short, elapsed time = 51, expected time = 180 20231013-05:25:06 Info Tile 0 : Prime Client Time 1697174705 UTC : Client0 Time 1697174702 UTC :: TimeErrors : 0 :: Clock Skew 3 20231013-05:25:08 Info Tile 1 : Prime Client Time 1697174707 UTC : Client1 Time 1697174705 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:10 Info Tile 2 : Prime Client Time 1697174709 UTC : Client2 Time 1697174707 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:13 Info Tile 3 : Prime Client Time 1697174711 UTC : Client3 Time 1697174709 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:15 Info Tile 4 : Prime Client Time 1697174714 UTC : Client4 Time 1697174711 UTC :: TimeErrors : 0 :: Clock Skew 3 20231013-05:25:17 Info Tile 5 : Prime Client Time 1697174716 UTC : Client5 Time 1697174714 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:19 Info Tile 6 : Prime Client Time 1697174718 UTC : Client6 Time 1697174716 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:22 Info Tile 7 : Prime Client Time 1697174720 UTC : Client7 Time 1697174718 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:24 Info Tile 8 : Prime Client Time 1697174723 UTC : Client8 Time 1697174721 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:26 Info Tile 9 : Prime Client Time 1697174725 UTC : Client9 Time 1697174723 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:28 Info Tile 10 : Prime Client Time 1697174727 UTC : Client10 Time 1697174725 UTC :: TimeErrors : 0 :: Clock Skew 2 20231013-05:25:32 Info Error: Tile 2 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 3 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 1 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 6 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 7 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 4 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 8 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 5 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 9 Weathervane results number not correctlyidentified 20231013-05:25:32 Info Error: Tile 10 Weathervane results number not correctlyidentified Thanks, Wen

Re: two questions in vmmark test

xiaolei_xue — Thu, 12 Oct 2023 23:25:58 GMT

Hi RebeccaG

The vCenter has two datacenter for two tests.

hosts 198.168.55.112 and .113,are the first test suts, 198.168.55.68 and .69,are the first suts' clients.

.111 and 110 are the second test suts and .67 and .66 are the suts' cliemts.

The two tests use different storage but the same vCenter.

If it affects the test.

Thank you ,

Wen

Re: two questions in vmmark test

RebeccaG — Thu, 12 Oct 2023 22:14:23 GMT

Hi Wen, yes I successfully unzipped your file using 7-zip, thank you.

There are errors causing Weathervane to not run and all infrastructure operations (vmotion, svmotion, xvmotion, deploy) are failing.

I think you have some issues with your shared storage and possibly vCenter Server.

The log VC-CollectEvents.csv shows many storage connectivity errors on hosts 198.168.55.112 and .113, 198.168.55.68 and .69, and these hosts are in SUTCluster and ClientCluster respectively. However, VC-ListHosts.txt shows only .111 and 110 in your SUTCluster and .67 and .66 in ClientCluster. Were these other hosts maybe in maintenance mode? Either way these storage errors could be affecting your vmmark run.

I would recommend you remove any unused hosts / hosts with errors from your VMmark clusters.

Your file VC-ListLUNs.txt also shows an error with the vmotion LUN:

Name : Free Space : Capacity : Uncommitted : Type : Multiple Host Access : Maintenance Mode
VMotion : 7819GB : 7821GB : ERROR : VMFS : true : normal

Using the vCenter UI, manually vmotion, storage vmotion or combined vmotion and storage vmotion a VMmark VM from one host to the other host in the SUT cluster.

Does it complete successfully? If not, watch the error that vCenter Server returns. You will need to fix this before you can run VMmark.

Thank you and please let me know if you have any further questions,

Rebecca

Re: two questions in vmmark test

xiaolei_xue — Wed, 11 Oct 2023 22:44:48 GMT

Hi RebeccaG， Because I failed to upload the package “Results_20231011065120_test1_2.zip”directly for Authentication fails,so I compressed the package again by volume. Can you use 7-zip to decompress the three packages together? Thanks, Wen

Re: two questions in vmmark test

RebeccaG — Wed, 11 Oct 2023 21:55:42 GMT

Hi Wen, I received three files from you:

Results_20231011065120_test1_2.zip.001,Results_20231011065120_test1_2.zip.002,Results_20231011065120_test1_2.zip.003.

I'm having difficulty extracting the files. Can you please use a command like this on the Prime Client to zip your VMmark results files?

zip -r Results_20190326174141.zip Results_20190326174141

Thank you,

Rebecca

Re: two questions in vmmark test

xiaolei_xue — Wed, 11 Oct 2023 13:40:20 GMT

HI RebeccaG， The vmmark test failed in the last test ,and I upload the log on the ftp. the the directory name is 'Wen',and the filenames are Results_20231011065120_test1_2.zip.001,Results_20231011065120_test1_2.zip.002,Results_20231011065120_test1_2.zip.003. I can upload them together while it always fails for Authentication Thans， Wen

Re: two questions in vmmark test

xiaolei_xue — Wed, 11 Oct 2023 06:29:07 GMT

Hi RebeccaG, Thank you for your reply. Please help me to reslove the problem as below. ===== XML Information ===== File: /root/VMmark3/xml/generic_functions.xml, Machine: local Line 141: Error in element type "script". ===== Python Error Information ===== com.ibm.staf.service.stax.STAXPythonEvaluationException: Traceback (most recent call last): File "", line 2, in NameError: name 'ErrorMsgs' is not defined ===== Call Stack for STAX Thread 64 ===== Thanks, Wen