RebeccaG's Posts

Thank you Oporras for uploading your results files. These are the VMmark storage infrastructure operations VMs: (quoted from VMmark3.properties file) // SVmotion/Workloads = Standby // XVmotion/Workloads = DS3WebA Please make sure that these VMs for every tile are located on a different datastore from ones you specified in VMmark3.properties: SVmotion/SVMotionLUNs = datastore1,datastore1 XVmotion/XVMotionLUNs = datastore1,datastore1 You can actually use the same datastore for all infrastructure operations as shown above, but they can also all be different.   The error you are seeing is because the workload VM in question is present on the same datastore as specified in SVMotionLUNs or XVMotionLUNs. It needs to be on a different datastore. This is why VMmark 3 requires at least 2 shared datastores across the SUT cluster.   In file Results_20231029035855_tile0, you have  SVmotion/SVMotionLUNs = DS_CLIPTEST01,DS_CLIPTEST02   You stated:  DS_CLIPTEST01 >   Standby0    Standby0 is on DS_CLIPTEST01, that is the source of the problem. Please migrate Standby0 to a different datastore or specify a different datastore in SVmotion/SVMotionLUNs.   In Results_20231029031758, SVmotion/SVMotionLUNs = DS_AMSA_QA_DEV,DS_BASESWIN_QA_01 You stated: DS_AMSA_QA_DEV >   Standby0  Same problem.   Also, I did not do a comprehensive review of your setup but I wanted to mention a couple of things I noticed: 1. Make sure you do not run VMmark on a production environment. There were many other VMs on your VMmark cluster.  From the VMmark 3 User's Guide, "NOTE VMmark, and the benchmark hardware and software components (vCenter, ESXi, and the VMmark template, clients, workload virtual machines, systems under test, storage, etc.) must be used only in an isolated test environment. None of the benchmark hardware or software components should ever be connected to a production network or transitioned to a production environment." Among other things, when you run VMmark it will create terrible performance for any production applications and produce invalid VMmark 3 results. 2. You have specified the same cluster for both the VMmark 3 Client cluster and SUT cluster. These need to be separate clusters and physical machines. You can still run VMmark with this configuration, for example, for functional testing, but it will produce invalid performance results.   Thank you, Rebecca

Hi oporras, VMmark is typically used to run as many tiles as possible in a cluster and still achieve a passing score. To find a rough estimate of how many tiles your cluster will support, you can look at the VMmark results page. Try filtering results by the total number of cores you have in your cluster. You may also be able to find a published VMmark result which uses the same or similar processor to your cluster, and extrapolate the number of tiles. You are also thinking about the memory and storage requirements for VMmark, that is good. Check the "System Requirements" on the VMmark website https://www.vmware.com/products/vmmark.html as it lists the amount of memory and storage needed for each tile. After you provision VMmark 3 with the tiles you calculated, see whether the run passed or failed (see the VMmark User's Guide for details).  If it passed, you need to provision more tiles so you can find the largest number of tiles that passes. If it failed you need to run VMmark 3 with one less tile. Continue until you have a passing run.

Hi oporras, VMmark 3 is typically used to run as many tiles as possible in a cluster and still achieve a passing score. To find a rough estimate of how many tiles your cluster will support, you can look at the VMmark results page. Try filtering results by the total number of cores you have in your cluster. You may also be able to find a published VMmark result which uses the same or similar processor to your cluster, and extrapolate the number of tiles. You are also thinking about the memory and storage requirements for VMmark, that is good. Check the "System Requirements" on the VMmark website https://www.vmware.com/products/vmmark.html as it lists the amount of memory and storage needed for each tile. After you provision VMmark 3 with the tiles you calculated, see whether the run passed or failed (see the VMmark User's Guide for details).  If it passed, you need to provision more tiles so you can find the largest number of tiles that passes. If it failed you need to run VMmark 3 with one less tile. Continue until you have a passing run.

Hi Wen, this is great. I see you resolved your earlier problems and are running 10 tiles with no setup issues. Number_of_Compliance_Issues (identified by '*' or '+') : 1 Issues Found : Tile4-weathervaneauction-p1 Median_Phase : p1 This is a Quality of Service (QoS) compliance issue. It means that the VMmark run failed to meet quality of service (i.e. Application Latency) requirements due to resource contention. This is normal when you are running a large number of tiles as eventually the QoS will fail due to storage contention, CPU contention, or other resource constraints. The failing tile, tile 4 is just barely failing. A passing score is less than 1. In VMmark3-graphs.html, it shows WeathervaneAuction has two spikes in QoS during the middle of the run. This is why the QoS failed. If you can resolve the cause of this poor performance, you might be able to pass a VMmark run at 10 tiles or more. You could also run VMmark again, and might be able to pass if you get lucky (slightly better performance from your environment). Thank you, Rebecca

Hi Wen, It should be ok if those hosts are in a different datacenter with same vCenter Server. For the new error, please zip your VMmark results file and if small enough, attach to this forum thread. If the zip file is too large please put it on ftpsite like last time. Also please try to manually vmotion or storage vmotion your VMmark VMs as it is failing in your environment. Also please check why your vMotion LUN shows error from my last post, it is the column 'uncommitted' which shows the error. Your file VC-ListLUNs.txt also shows an error with the vmotion LUN: Name : Free Space : Capacity : Uncommitted : Type : Multiple Host Access : Maintenance Mode VMotion : 7819GB : 7821GB : ERROR : VMFS : true : normal Second, all infrastructure operations (vmotion, xvmotion, svmotion, deploy) .wrf files show the response "FAIL". This error from vCenter Server means the operation cannot complete. To troubleshoot this, replicate the activity manually: Using the vCenter UI, manually vmotion, storage vmotion or combined vmotion and storage vmotion a VMmark VM from one host to the other host in the SUT cluster. Does it complete successfully? If not, watch the error that vCenter Server returns. You will need to fix this before you can run VMmark.

Hi Wen, yes I successfully unzipped your file using 7-zip, thank you. There are errors causing Weathervane to not run and all infrastructure operations (vmotion, svmotion, xvmotion, deploy) are failing. I think you have some issues with your shared storage and possibly vCenter Server. The log VC-CollectEvents.csv shows many storage connectivity errors on hosts 198.168.55.112 and .113, 198.168.55.68 and .69, and these hosts are in SUTCluster and ClientCluster respectively. However, VC-ListHosts.txt shows only .111 and 110 in your SUTCluster and .67 and .66 in ClientCluster. Were these other hosts maybe in maintenance mode? Either way these storage errors could be affecting your vmmark run.  I would recommend you remove any unused hosts / hosts with errors from your VMmark clusters. Your file VC-ListLUNs.txt also shows an error with the vmotion LUN: Name : Free Space : Capacity : Uncommitted : Type : Multiple Host Access : Maintenance Mode VMotion : 7819GB : 7821GB : ERROR : VMFS : true : normal Second, all infrastructure operations (vmotion, xvmotion, svmotion, deploy) .wrf files show the response "FAIL". This error from vCenter Server means the operation cannot complete. To troubleshoot this, replicate the activity manually: Using the vCenter UI, manually vmotion, storage vmotion or combined vmotion and storage vmotion a VMmark VM from one host to the other host in the SUT cluster. Does it complete successfully? If not, watch the error that vCenter Server returns. You will need to fix this before you can run VMmark. Thank you and please let me know if you have any further questions, Rebecca  

Hi Wen, I received three files from you:  Results_20231011065120_test1_2.zip.001,Results_20231011065120_test1_2.zip.002,Results_20231011065120_test1_2.zip.003. I'm having difficulty extracting the files. Can you please use a command like this on the Prime Client to zip your VMmark results files?  zip -r Results_20190326174141.zip Results_20190326174141 Thank you, Rebecca

Hi Wen, I apologize I didn't get to this October 5, and VMware had a company holiday October 6. Yes, I see from your directory, you have a completed VMmark run which is too large to upload to the forum. Here is how to send me the results. Zip the results folder (e.g. .zip, .tgz, .rar), and please visit http://ftpsite.vmware.com user: inbound password: inbound Click New Directory - F7 and choose a new directory name. Click OK. Click Change Directory and input your new directory name. Click Change. Click Upload to select your zipped files. Finally, send me the directory name and filenames. Thank you, Rebecca

Hi Wen, Problem #1 There are many things that can cause this error. Please zip your VMmark 3 results file and upload it to a reply on this post, thank you. This will help me see the problem. Problem #2 This is not a problem, you can ignore this warning. It is a warning that occurs when running on vSphere 8.0 and later.

Hi Patelch, Yes, this is a known issue that recently came to our attention. We are currently working on an update to the VMmark User’s Guide that will allow you to set up passwordless SSH on ESXi 7.0 U3d. As soon as possible we will be releasing this update to the VMmark User's Guide. Thank you, Rebecca

I apologize for the delay in response. In looking at the three tile result, the Deploy error occurred 1 hour into the 3 hour run. At the beginning of the run, a deploy operation completed correctly on all of your hosts. The second time Deploy tried to run on esxi04-ehc-ehcdc.com, the error ocurred "ERROR  DeployVM0 Iteration 3 Unable to Create VM : Msg An error occurred while communicating with the remote host." (as reported in Deploy-t0.csv) The real issue is that the Deploy VM was unable to be created, and it looks like the hot add error occurred because the VM wasn't created. Try rerunning 3 tiles and see if the error recurs. It might not recur. If it does, you need to know whether the error occurs only on host esxi04, and when during the run it occurs. You might also want to check network connection between the vCenter Server and host esxi04, as it returned "an error occurred while communicating with the remote host".  Also, did the increased MiscInfoTimeOut help?

To run without infrastructure operations, see in the VMmark User's Guide "Running a Subset of the Workloads (a Partial Tile). You'll uncomment the variable "InfrastructureList" from VMmark3.properties and remove all infrastructure workloads. However, this will result in a benchmark run that does not generate a score (Score_4_Tile_Test.txt) as it is not a compliant configuration.  For the guestinfo failure, it looks like the guestinfo collection was unable to complete before timing out. (these files are in your results folder in guestinfofiles/DS3DBN-Guestinfo.txt) In your VMmark3.properties file, try changing MiscInfoTimeOut = 600 or even larger and rerunning. Please upload the run with the error "Unable to Update DeployVM0..." when you get a chance. Thank you, Rebecca

Hi Nimo, Thank you so much for uploading your results files; with it I was able to find the problem. Three main things you will need to fix: The hosts files for some of the VMmark VMs contain the IP addresses of the wrong tile VMs. E.g. ElasticLB1 should have hosts file entries for tile1. These were also incorrect for all tile 3 VMs. The VMs’ hosts files are automatically populated by the VMmark harness when creating additional tiles. It indicates maybe you tried to clone the tiles manually or altered the hosts files. To fix this, delete tiles 1, 2 and 3. Recreate them following “Creating Additional VMmark Tiles (Tiles 1 through n)” in the VMmark User’s Guide.   You are using static IPs on the 10.61.187 subnet. It’s recommended that VMmark VMs use a private network and the prime client is set up with two virtual NICs, one on the private network and one on the public network for administrative access. Most users have trouble getting enough static IPs allocated for all the VMmark VMs as 19 Ips are required per tile. If you decide to keep operating on the 10.61.187 subnet, when you recreate tiles 1, 2 and 3, in VMmark3.properties, set your ProvisioningIPstaticStart = 10.61.187.126. Tile 1 will be provisioned in the correct range as the VMmark harness accounts for tile 0.   3. There is also an IP address conflict between the Deploy VM (Deploy/DeployVMinfo = DeployVM0:10.61.187.190) and DS3DB2 which has the same IP. You need to set DeployVMinfo to a static IP that is not in the range you specified for ProvisioningIPstaticStart. E.g. use 10.61.187.125. This issue would be resolved if you configure VMmark using an internal network instead of an external one. Thank you and please post here if you have any additional questions.

Hi sfit1, thank you for providing more information about your testing environment and the results .zip. Unfortunately, I believe your storage latency and/or bandwidth is too slow to support the VMmark workloads. VMmark typically requires all-flash storage, or a hybrid (flash and HDD) vSAN setup. You can see which storage was used for published VMmark results at https://www.vmware.com/products/vmmark/results3x.html.   You can see in VMmark3-Graphs.html that spikes in QoS latency occur at the beginning of the run. This happens for all workloads except vMotion which is not storage-dependent. This probably indicates that storage cacheing is occurring at the beginning of the run. By later in the run once most hot data is cached, the QoS is much lower. But the slow performance at the beginning still causes VMmark to fail QoS requirements. Unfortunately you will need to use a faster storage solution to test this environment. If you can add some SSDs to your FC SAN and place the Weathervane VMs there, that would be a good place to start.