RebeccaG's Accepted Solutions

Hi oporras,  Please don't include spaces or special characters in vSphere names (usernames, datacenters, clusters, etc.). You can see this is in the VMmark User's Guide under "Configure vCenter Server". Once you have fixed that, if you are still having a problem, please zip the VMmark results folder and upload it as an attachment to this forum thread. This will help me troubleshoot the issue. Thank you, Rebecca  

To sum up, the reason you were getting 0 score is a load-related issue where the Deploy wasn't finishing. If you compare the 6-tile result and the 8-tile result, you can see your SVMotion, XVMotion and Deploy latencies have doubled. This is where to look in Score_8_Tile_Test_NC.txt: That would implicate the storage, or I suppose the vCenter Server is also involved in the operation. I'm assuming the environment was the same between both runs. The timestamp on both the 6-tile and 8-tile runs is similar, April 13. Did you run the 8-tile run first and then the 6 tile run second? All the QoS looks really good on the 6-tile run, and terrible on the 8-tile run.

The number of simultaneous infrastructure operations is not determined by the amount of shared storage. It's determined by the number of hosts and the number of tiles. The number of simultaneous infrastructure operations is half of the smaller of either the number of hosts or the number of tiles, rounded down to an integer. It is a built-in parameter of the benchmark which depends on the number of hosts and tiles you are running. VMmark requires at least 2 shared vSphere datastores. This is to allow storage vmotion and xvmotion from one datastore to the other. So it's also OK to have only two shared datastores. For each simultaneous infrastructure operation, you need to specify the target datastore in VMmark3.properties. The way to do this looks like: SVmotion/SVMotionLUNs = datastore1,datastore2 and also for parameters XVmotion/XVMotionLUNs, Deploy/DeployLUNs. So with 4 simultaneous infrastructure operations, specify four datastores. These can all be the same datastore, or different datastores.

Hi there, when you did a 'staf ping' from the prime client to the ElasticAppA0, ElasticAppB0, and ElasticDB0, did you use the hostname that is present in the prime client's hosts file? And when you ssh into ElasticAppA0, ElasticAppB0, and ElasticDB0, 'staf primeclient ping ping' returns 'pong'? Have you altered any of VMs' hosts files since provisioning? Try looking in the troubleshooting section of the VMmark User's Guide, "STAF and STAX Issues". Since the staf ping is not working, the issue is either going to be network related, or that STAF is not running correctly on the Elastic VMs. I think you've already understood this by trying to reboot the VMs in question, but try taking a look at that section if you haven't already.

Thank you for such a complete answer. I’m glad to hear that you were able to get 3 tiles. Each tile is a unit of load. The simple answer is that your system under test doesn’t support 4 tiles of load at a passing quality of service. From what you’ve described, you have more than enough CPU and processing power to go around. Most likely, you have a storage bottleneck from the VSAN HDDs. There are a number of published VMmark results that are VSAN. VMmark 3.x Results  Look for Quanta Cloud, Sugon, and Supermicro. Check to see whether they are all-flash or hybrid. I doubt network or client CPU are a problem for you. A. Yes, it is compliant to place the VMmark VMs on any shared storage, as long as the VMs are still part of the VMmark cluster and the VMs can be storage vMotioned to the SvMotion target LUN and vMotioned to any host in the cluster. B. I don’t think so. You could try a faster processor for the Client hosts possibly. You can still get a passing VMmark run if you increase the clients’ # of vCPU, but the result will not be compliant, in that it cannot be published on VMmark.com.

Hi njuerect1, Yes, you are on the right track. Each asterisk (*) is a Quality of Service failure (QoS) for some application during the run. In Score_N_Tile_Test.out, if you look at the section "Tile_0_QoS(ms):" you will see asterisks next to some of the values. These are the application latencies in milliseconds measured by the harness. You are correct that each application has its own minimum QoS requirement. Yes, application latencies are generally the response time between when the client issues a transaction request, and when that request is completed. You said that the "*" appeared only once after 8-10 VMmark tests. This usually can happen if the system under test is pushed nearly to its maximum load level and is experiencing resource contention of some kind (CPU, memory disk, etc.). Some VMmark tests will pass (no "*") and others might fail (there is a "*") even if there are no changes to the testbed configuration. This is actually a good sign if you are trying to run as many tiles as possible on your VMmark testbed. It means you are testing the system at its maximum level of load. Please let me know if you have any further questions.

Hi Alec, First I apologize for the delay in response. We are having an issue with the VMmark community notifications so I just saw your post now. When a VMmark workload is not running, and therefore not producing output, VMmark cannot create the score file.  The harness processes the .wrf files and if it cannot parse these, you get the result "Error: could not resolve start-time." In your case, the OlioWeb1.wrf is causing the error. Its output is "Error: Could not find or load main class radlab.rain.Benchmark". From what I have seen, this is caused by a corrupted file in the C:\vclient\olioweb directory. On your client1, please replace the C:\vclient\olioweb directory with a fresh copy from the prime client or from client0 and rerun. Please feel free to post back here with any future issues you may see. Thank you, Rebecca

OK, first, go to the VMmark Benchmarking Guide and find the section "Prepare the DVD Store 2 Database". Right above this is a step #12 that begins "Create the web user...". See attached image. On your DS2DB VM, complete all steps in this section. Then rerun VMmark. If you are still seeing the same error in ConfigDS2DB_0.txt, please delete the DS2DB and redeploy the DS2 database .ova. Once you deploy the .ova, you will need to complete the steps in the Benchmark guide section "Prepare the DVD Store 2 Database" in which you will run several scripts including Install_DVDStore_vmmark.pl. Please let me know if you have any further questions. Thank you, Rebecca

Hi Suresh, This is happening because the PATH environment variable is not resolving the location of the cygwin binaries 'ls', 'grep', 'wc', etc. Check your PATH environment variable on the prime client and make sure it includes the path such as "C:\cygwin\bin;". You mentioned earlier you had reinstalled cygwin on the prime client. It doesn't matter whether you use 32 or 64-bit cygwin as long as it is installed to location C:\cygwin. Also, in VMMARK2.CONFIG, please change ###DEBUGFLAG=0 to DEBUGFLAG=1 to enable debugging mode. This will help provide us more information about your environment.

Hi Suresh, Actually, the prime client does not generate the wrf files. The tile's client, client0, generates the wrf files. The prime client runs the harness and collates the results from each client into the results file. The program that pings standby is located on client0, and it's having trouble starting because of the wrong Java bitness on client0. This program writes to standby0.wrf. Your Standby VM is likely fine. Rebecca

It looks like, like jamesz08 mentioned, 8.3 naming is disabled on your client, which you will need to enable. In the results that you posted, no short name for LoadGenCmd.exe was listed (e.g. LOADGE~2.EXE). We would have expected that line to read: 04/21/2008     05:39 AM     59,472 LOADGE~2.EXE LoadGenCmd.exe Another difference you can see is that the date and time need to match the format MM/DD/YYYY and the 12-hour clock. Make sure the date and time format used by all Windows-based systems (that is, all clients and the vCenter server) is the USA standard (that is, month/day/year). Please let us know if you have any issue with these reconfigurations. Thanks, Rebecca

This metric shows the amount of free memory you have on your client. Whether this number is higher than normal or not depends on how much RAM you have on the physical client. Is the free memory higher than in your past runs? VMmark requires that you have at least 4GB of RAM for each client (physical or virtual), so you may have more than that. Generally I see that about 1GB of memory on the client is consumed during a VMmark run, so for a 4GB memory client, 3000 megabytes will be free according to this metric. If you attach your Results folder to this post, I can check your run to see whether LoadGen/ Mailserver is functioning normally. Thanks, Rebecca

The script automatically uncomments all values in cygserver.conf, lines like for example kern.srv.cleanup_threads 2. In the files you showed, the values were already uncommented so there's no change. Your cygserver.conf is correct.

Thank you, the System screen shows that your mailserver domain is currently "your.company.com". The domain needs to be "maildomain0.your.company.com". To fix this, you cannot simply change the name of the Mailserver VM. You will need to go back to the Mailserver template you created earlier, then reclone your Mailserver VM and reconfigure it following the instructions in "Prepare the Mail Server Virtual Machines" in the VMmark Benchmarking Guide. At the step "Name the Forest Root Domain" make sure you enter maildomain0.your.company.com. When you are finished, in VMMARK2.CONFIG, specify: MailServer/MailDomains="maildomain0" MailServer/MailQualifier="your.company.com" Thanks, Rebecca

Hi Jones, This behavior is by design. The Score_N_Tile_Test.out file, which calculates the VMmark score, is created only when all workloads are running, that is, in VMMARK2.CONFIG: WORKLOADLIST="Standby MailServer OlioWeb OlioDB DS2WebA DS2WebB DS2WebC DS2DB" and INFRASTRUCTURELIST="Deploy Vmotion SVmotion" Since you did not run Mailserver (and in the provided VMMARK2.CONFIG you exclude Deploy also), Score_N_Tile_Test.out was not created. However, you can manually run the script that produces Score_N_Tile_Test.out, and create a Score file on an already finished run by manually excluding the Mailserver workload which was not running at the time. In the VMmark Benchmarking Guide, see "Obtain Partial Results for a Failed or Non-compliant Run" for how to do this. However, you will not be able to get a "VMmark Score" which requires all workloads to run. Please let me know if you have any further questions. Thanks, Rebecca

"'Error : vm03.nubav.thinkon.net Failed SSH Check" means that passwordless SSH is not set up properly on the hosts in your VMMARK cluster. Actually vm07, vm08, vm09 and vm10 are not the hosts in your VMmark cluster. In your VMMARK2.CONFIG, you have listed VCServerHOSTNAME 10.1.80.15, and VCServerCLUSTER prodcl101. If you look at the STAX message log you will see that the hosts in the VMmark cluster are: VMmark 2 : Hosts In Cluster : ['vm03.nubav.thinkon.net', 'vm04.nubav.thinkon.net', 'vm05.nubav.thinkon.net', 'vm06.nubav.thinkon.net', 'vm19.nubav.thinkon.net', 'vm20.nubav.thinkon.net', 'vm21.nubav.thinkon.net', 'vm18.nubav.thinkon.net', 'vm22.nubav.thinkon.net', 'vm02.nubav.thinkon.net'] If you were expecting vm07, vm08, vm09, vm10, check that the right VCServerHOSTNAME and cluster name is listed in VMMARK2.CONFIG.

Nice diagram! I see you have 4 LUNs for VMmark tiles, plus 1 LUN for DeployTemplate. In my opinion, 9 disks per LUN, with 2 tiles per LUN is probably not fast enough storage. It would be adequate for 1 tile per LUN. If you have not already, try this: 1. Run 5 tiles with 1 tile on each LUN (1-4) except for 1 LUN holding two tiles. (for example, tile 0 and tile 4 are both on LUN1) 2. In Score_5_Tile_Test, look at the QoS scores for the tiles which share the same LUN. Are the QoS scores higher than the other tiles? If so, this could indicate a storage bottleneck. Also try changing Deploy/DeployLUNs="vm_lun1" to Deploy/DeployLUNs="vm_lun0". You should generally use different LUNs for DeployLUN and tile VMs so Deploy does not impact the workload VM performance. One other thing that can cause unpredictable performance is if the server's power management setting in the BIOS is set to 'balanced' (power saving) or similar; you should always set to OS control. Thanks, Rebecca