dmorse's Accepted Solutions

Here are the issues I see: These two parameters should not be the same: vCServerVMmark3Cluster = ClientCluster vCServerClientCluster = ClientCluster Per the User's Guide, "VMmark requires two vSphere clusters, one for the workload VMs (referred to as the SUT cluster) and one for the Client VMs (referred to as the Client cluster). These parameters are not set, and must be shared storage: SVmotion/SVMotionLUNs = XVmotion/XVMotionLUNs = This parameter is set, but please make sure it is shared storage (i.e., Fibre Channel, iSCSI, NFS, etc.): Deploy/DeployLUNs = datastore1 The VMmark 3 benchmark output states "1 Active VMmark3 Hosts" (IP address of 192.168.1.253). The User's Guide states "the benchmark requires a minimum of two ESXi hosts" for the System Under Test cluster. The two settings/values below are incompatible with each other: // For provisioning Tile 1 - N, Tile0 is used as the source and this should be the keyword 'tile0' : Ex 'ProvisioningSource = tile0' ProvisioningSource = tile0 // int ProvisioningNumTiles : 1 or more tiles : Default 1 : If ProvisioningSource is a template this is automatically set to 1 tile. ProvisioningNumTiles = 1 From the output you provided, it appears you have already successfully provisioned 1 tile (tile0).  If you wish to provision a second tile, you would set ProvisioningNumTiles to 2. But as I said earlier, the larger issue is that you need two separate clusters, and the SUT cluster needs at least two ESXi hosts with shared storage.

Hi Ajay, In your properties file, try uncommenting these lines and setting them higher than the default values: // int ProvisionPingMaxTries : Default 24 : Specifies the number of ping attempts before failing. Each ping attempt waits 30 seconds. // ProvisionPingMaxTries = 24 // int ProvisionPingIpSeconds : Default 600 : Specified the number of seconds to wait for a VM to get an IP address. // ProvisionPingIpSeconds = 600 // int ProvisioningMaxSimultaneousOperations : Default is the number of workload VMs being operated on. This can be used to slow down the deployment requests. // ProvisioningMaxSimultaneousOperations = For example: ProvisionPingMaxTries = 50 ProvisionPingIpSeconds = 1200 ProvisioningMaxSimultaneousOperations = 3 Also, I noticed a couple of other things: - You are using the same cluster (custom_cluster) as your System Under Test (workload) cluster and client cluster: // String vCServerVMmark3Cluster : Default empty : Ex vCServerVMmark3Cluster = Cluster1 vCServerVMmark3Cluster = custom_cluster // String vCServerClientCluster : Default empty : Ex vCServerClientCluster = ClientCluster1 : This is the cluster of hosts running your // Client VMs and is where provisioning will place new client VMs. vCServerClientCluster = custom_cluster This is not supported, please split them out to separate clusters. All of your VM types (clients and workloads) are being stored on one datastore named "datastore".  This could be overwhelming your storage, especially if it's not all flash.  Can you describe if you have a SAN, how many disks you have, what type they are, confirm it's shared storage, etc.? Thanks, David

Hi @jiangjing123455 , I am not sure why only the first test passes. If you shut down all 5 tiles using this command from the prime client (make sure you're in the /root/VMmark3 directory first): java -jar tools/VMmark3Service.jar -c VMmark3.properties -m tilePower -tiles 0 Then start all 5 tiles using this command: java -jar tools/VMmark3Service.jar -c VMmark3.properties -m tilePower -tiles 5 And then try another 5-tile run, does that pass?

Hi @jiangjing123455 , These are only warning messages, and are not unusual to see in any VMmark3 run, even at 1 tile (I see them in my environment too). Note that after the exceptions are listed, you see a "Run_Is_Compliant" message, so no worries.

Something is definitely wrong with the tile1 and tile2 VMs, particularly the client1 and client2 VMs. If you look at the configfiles/Client[0,1,2]_hosts.txt, you'll see that they all point to tile0 SUT VMs.  This is why I see errors like this in the STAX_Job_6_User.log: 20210630-12:23:51 Info Weathervane Auction Abort : Tile 1 : Weathervane Auctio n Terminated Unexpectedly 20210630-12:23:51 Info DS3WebA Abort : Tile 1 : ds3webdriver Process Terminate d Unexpectedly 20210630-12:23:54 Info DS3WebB Abort : Tile 1 : Iteration 0 ds3webdriver Proce ss Terminated Unexpectedly 20210630-12:23:59 Info DS3WebC Abort : Tile 1 : Iteration 0 ds3webdriver Proce ss Terminated Unexpectedly 20210630-12:24:51 Info Weathervane Auction Abort : Tile 2 : Weathervane Auctio n Terminated Unexpectedly 20210630-12:24:52 Info DS3WebA Abort : Tile 2 : ds3webdriver Process Terminate d Unexpectedly 20210630-12:24:55 Info DS3WebB Abort : Tile 2 : Iteration 0 ds3webdriver Proce ss Terminated Unexpectedly 20210630-12:24:58 Info DS3WebC Abort : Tile 2 : Iteration 0 ds3webdriver Proce ss Terminated Unexpectedly I suggest you delete all tile1 and tile2 VMs (both SUTs and clients), and re-run the provisioning to re-create them.  Hopefully then client1 and client2 will have valid /etc/hosts files that point to tile1 and tile2 instead of tile0.

On a seperate note, if I want to run only "Infrastructure Operations", Can I set the "Workload List" to empty value and start the run? I wasn't sure about this myself, so I tried a run with no workloads and just infrastructure operations, and it failed when trying to produce a score.  So no, you need to run 1 or more workloads for a valid score to be produced.

Hi mathieugont​ Sorry you're still having issues.  A few things I want to point out: It looks like you have created a cluster for each ESXi host, and the first cluster, ClusterESXi001, has 1 host, but all of the tile 0 VMs on it.  This will not work for multiple reasons: The VMmark3 benchmark is designed to vMotion, storage vMotion, etc. VMs from 1 host to another within the same cluster. The benchmark is designed to only work with a single client ESXi host cluster (ClusterESXi005 in your case), and a single server ESXi host cluster (which you have broken out into ClusterESXi001,2,3,4). Creating shared storage such as NFS, while it may allow the benchmark to complete, may not give you well-performing scores without high-performing storage (i.e. SSD arrays).  Can you share the array details? All of that aside, though, there seems to be a fundamental networking problem causing the "Guest Info failures".  If you SSH into your PrimeClient VM, are you able to ping the tile0 VMs (i.e. 'ping Standby0, 'ping DS3WebA0')?

The problem is that the DS3 database never completely got built.  Per the below output from ConfigDS3DB_0.txt, the reviewshelpfulness table failed the check (this is the longest step in the database creation process): Checking the contents of Tables = reviewshelpfulness ERROR:  DS3 value check failed. ============================================================================ -------------------------------------------------------------------------- FAIL:  DS3 is either not built completely or not in initialized state. -------------------------------------------------------------------------- 04:50:18 PM: DS3 checker script finished.. 04:50:18 PM: DS3 DB check failed.. Per the User's Guide on page 50: "Wait for the DS3DB0 creation process to complete; depending on your environment, this can take 12 hours or more. To determine if it’s complete, follow these steps: a     In a terminal window on the prime client, run the following command: ssh <DS3DB0-IP-address> ‘tail -f /ds3/VMmark3-DS3installoutput.txt’ (where <DS3DB0-IP-address> is the IP address of the DS3DB0 VM, which can be found in the hosts-stub.txt file referenced above). b      Watch for the following text: VMmark3 DS3DB Setup Completed : Rebooting" Unfortunately, I think you'll need to re-provision your tile 0 again, and wait until you see the "DS3DB Setup Completed".

Hi, could you please perform the following steps so we can troubleshoot further? 1. From your primeclient VM, attach the VMmark3.properties you used to run the benchmark (/root/VMmark3/VMmark3.properties) 2. From your primeclient VM, ZIP and attach the configfiles subdirectory from your most recent run: cd /root/VMmark3/results/Results_2018* (note: make sure you choose the most recent failed run directory) zip -r /root/configfiles.zip configfiles Then attach the resulting /root/configfiles.zip 3. From your DS3DB0 VM, attach the output of these commands: cd /root/VMmark3/client/DS3DB ./DS3_checker.sh Thanks, David

Hello, No, a deploy operation should never take that long to complete.  I would guess there is a bottleneck (or a hardware issue?) within the vSAN environment. Can you post details of the vSAN hardware configuration?  Specifically, the speeds and types of HDDs/SSDs? Do you see storage-related alarms within your vCenter Server?

Please try to see if you can manually deploy your template.  Here are the steps: From the vSphere Web Client, right-click on your deploy template. Select "New VM from This Template". In the name box enter "DeployVM0". Select the correct DC location and click next. Select your cluster and click next. Select your LUN and click next. Click the box next to "Power on virtual machine after creation" so that it is enabled. Click the box next to "Customize the operating system" and click next. Select the "StandbyCust" specification and click next. Click Finish. Watch the deploy and customization from the vSphere Web Client. You should be able to see the VM power on and customize from the console without logging in. Wait about 5 minutes to see if all the cloning and OS specification processes have completed.  Then without logging into the newly created DeployVM, from the prime client, at a command prompt, type the following. staf DeployVM0 ping ping The proper response is PONG.