9 Replies Latest reply on Jun 7, 2019 1:07 PM by RebeccaG

    VMmark/Auction fails with RC=1

    mathieugont Novice

      Hi David

       

      Thanks for your advice.

       

      I thus setup an NFS I have mounted on each cluster of my datacenter (same absolute path on each node). See the print screens below.

      I had a close look the the property file, but I do not see how I can point my shared file-system.

      Moreover, VMmark/Auction fails with RC=1 which refer to an API issue (see the messages bellow).

       

      Any idea of what is going wrong? I am running out of ideas.

      Thanks.

       

       

      20190522-11:02:14 Info  Process: Client0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable doesnot exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:02:14 Info  Client0 : Get Guest Info did not run correctly

      20190522-11:02:14 Info  Process: AuctionLB0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable does not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:02:14 Info  AuctionLB0 : Get Guest Info did not run correctly

      20190522-11:02:14 Info  Process: localhost : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable does not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:02:14 Info  localhost : Get Guest Info did not run correctly

      20190522-11:02:14 Info  GuestInfo Results : AuctionLB0 :: Failed : Errors 0 : ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:02:14 Info  GuestInfo Results : Client0 :: Failed : Errors 0 : ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:02:14 Info  GuestInfo Results : localhost :: Invalid : Errors 1 : ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:03:14 Info  Process: AuctionWebA0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variabledoes not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:03:14 Info  AuctionWebA0 : Get Guest Info did not run correctly

      20190522-11:03:15 Info  GuestInfo Results : AuctionWebA0 :: Failed : Errors 0 :ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:04:15 Info  Process: AuctionWebB0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variabledoes not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:04:15 Info  AuctionWebB0 : Get Guest Info did not run correctly

      20190522-11:04:15 Info  GuestInfo Results : AuctionWebB0 :: Failed : Errors 0 :ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:05:15 Info  Process: AuctionAppA0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variabledoes not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:05:15 Info  AuctionAppA0 : Get Guest Info did not run correctly

      20190522-11:05:15 Info  GuestInfo Results : AuctionAppA0 :: Failed : Errors 0 :ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:06:15 Info  Process: AuctionAppB0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variabledoes not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:06:15 Info  AuctionAppB0 : Get Guest Info did not run correctly

      20190522-11:06:16 Info  GuestInfo Results : AuctionAppB0 :: Failed : Errors 0 :ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:07:16 Info  Process: AuctionDB0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable does not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:07:16 Info  AuctionDB0 : Get Guest Info did not run correctly

      20190522-11:07:16 Info  GuestInfo Results : AuctionDB0 :: Failed : Errors 0 : ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:08:16 Info  Process: AuctionMSQ0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable does not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:08:16 Info  AuctionMSQ0 : Get Guest Info did not run correctly

      20190522-11:08:16 Info  GuestInfo Results : AuctionMSQ0 :: Failed : Errors 0 :ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:09:16 Info  Process: AuctionNoSQL0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variabledoes not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

      20190522-11:09:16 Info  AuctionNoSQL0 : Get Guest Info did not run correctly

      20190522-11:09:17 Info  GuestInfo Results : AuctionNoSQL0 :: Failed : Errors 0: ErrorMsgs  : Warnings 0 : WarningMsgs

      20190522-11:09:47 Info  Guest Info failures for the following machines: 10 machines : Client0 AuctionLB0 localhost AuctionWebA0 AuctionWebB0 AuctionAppA0 AuctionAppB0 AuctionDB0 AuctionMSQ0AuctionNoSQL0   Error Messages Client0= :AuctionLB0= :localhost= :AuctionWebA0= :AuctionWebB0= :AuctionAppA0=:AuctionAppB0= :AuctionDB0= :AuctionMSQ0= :AuctionNoSQL0= :

       

        • 1. Re: VMmark/Auction fails with RC=1
          dmorse Enthusiast
          VMware Employees

          Hi mathieugont

           

          Sorry you're still having issues.  A few things I want to point out:

          • It looks like you have created a cluster for each ESXi host, and the first cluster, ClusterESXi001, has 1 host, but all of the tile 0 VMs on it.  This will not work for multiple reasons:
            • The VMmark3 benchmark is designed to vMotion, storage vMotion, etc. VMs from 1 host to another within the same cluster.
            • The benchmark is designed to only work with a single client ESXi host cluster (ClusterESXi005 in your case), and a single server ESXi host cluster (which you have broken out into ClusterESXi001,2,3,4).
            • Creating shared storage such as NFS, while it may allow the benchmark to complete, may not give you well-performing scores without high-performing storage (i.e. SSD arrays).  Can you share the array details?

           

          All of that aside, though, there seems to be a fundamental networking problem causing the "Guest Info failures".  If you SSH into your PrimeClient VM, are you able to ping the tile0 VMs (i.e. 'ping Standby0, 'ping DS3WebA0')?

          • 2. Re: VMmark/Auction fails with RC=1
            mathieugont Novice

            Hi David

             

            I followed your recommendations grouping all my nodes within a single cluster (4 hardware servers grouped in ClusterESXi_BDW), except one (in ClusterESXi_Mgr) hosting the VMware vCenter Server Appliance (named photon), and a personal VM used to access vSphere client remotely (internal config). See the print screen below.

             

            In ClusterESXi_BDW, all my VMs are hosted by the node 172.22.197.109, while my Client0 and my Prime Client are on the node 172.22.193.115.

            From my Prime Client, I can ping and passwordless ssh all the VMs.

             

            About a well-performing scores without high-performing storage (i.e. SSD arrays), the aim is to make it work first. Score and improvement will be treated in a second hand. For the moment, I need to run weathervane only, that is why in my property files, I set:

            WorkloadList = AuctionLB,AuctionWebA,AuctionWebB,AuctionAppA,AuctionAppB,AuctionDB,AuctionMSQ,AuctionNoSQL

            InfrastructureList =

             

            But as you can see in the zip file attached to this message, comment "InfrastructureList =" to get the default does not solve the problem.

             

            • 3. Re: VMmark/Auction fails with RC=1
              mathieugont Novice

              Quick additional comment

              • My attached files are without infrastructure VMs and the Auction only on a single tile. Is WMmark is designed to work it? If not, should I deploy a second tile on another hardware server?
              • Message are finally different when I add infrastructure VMs. 
              • 4. Re: VMmark/Auction fails with RC=1
                dmorse Enthusiast
                VMware Employees

                Hi mathieugont

                 

                I checked with the technical lead for VMmark, he recommended rebooting your PrimeClient VM and double-checking that there aren't any duplicate IP addresses in your network.  It looks like something aborted the Guest Info collection on the PrimeClient -- these kinds of messages are very unusual:

                 

                20190531-18:07:34 Info  Process: localhost : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable does not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

                20190531-18:07:34 Info  localhost : Get Guest Info did not run correctly

                20190531-18:07:34 Info  Process: AuctionLB0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable does not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

                20190531-18:07:34 Info  AuctionLB0 : Get Guest Info did not run correctly

                20190531-18:07:34 Info  Process: Client0 : Get Guest Info failed to start/complete. Returned: RC = 1, STAFResult = This variable doesnot exist: VMMARK/AuctionNoSQL/SKIPRESTORE

                20190531-18:07:34 Info  Client0 : Get Guest Info did not run correctly

                 

                Also -- if you want to just run Weathervane, that workload is an open-source project on github:

                GitHub - vmware/weathervane: Virtual-Infrastructure and Cloud Performance Benchmark

                 

                Weathervane itself has a run harness that might be easier to use than trying to run VMmark without the other workloads, but it's up to you.

                • 5. Re: VMmark/Auction fails with RC=1
                  mathieugont Novice

                  Thanks David

                   

                  Did you saw the same message appeared whatever the VM:

                  Returned: RC = 1, STAFResult = This variabledoes not exist: VMMARK/AuctionNoSQL/SKIPRESTORE

                   

                  RC=1 refers to API issues. Not a good sign ;-)

                  But seems AuctionNoSQL is the issue.

                   

                  Any comment on that?

                  • 6. Re: VMmark/Auction fails with RC=1
                    mathieugont Novice

                    David

                     

                    I am trying to run weathervane out of VMmark like you advised to me.

                    I followed the procedures described in section 3 and 4 of the manual. I needed to tune the scripts, but it the end, it does not work with the following message:

                     

                    Wed Jun 5 14:46:29 2019: Weathervane Version 1.0.21

                    Wed Jun 5 14:46:29 2019: Command-line parameters:

                    Wed Jun 5 14:46:29 2019: Run Configuration has 1 workloads.

                    Wed Jun 5 14:46:29 2019: Workload 1 has 1 workload-driver nodes

                    Wed Jun 5 14:46:29 2019: Workload 1 has 1 application instances.

                    Wed Jun 5 14:46:29 2019: Workload 1, Application Instance 1 configuration:

                    Wed Jun 5 14:46:29 2019:        1 ConfigurationManagers

                    Wed Jun 5 14:46:29 2019:        0 ElasticityServices

                    Wed Jun 5 14:46:29 2019:        1 CoordinationServers

                    Wed Jun 5 14:46:29 2019:        0 LbServers

                    Wed Jun 5 14:46:29 2019:        1 WebServers

                    Wed Jun 5 14:46:29 2019:        1 DbServers

                    Wed Jun 5 14:46:29 2019:        1 NosqlServers

                    Wed Jun 5 14:46:29 2019:        0 FileServers

                    Wed Jun 5 14:46:29 2019:        1 MsgServers

                    Wed Jun 5 14:46:29 2019:        1 AppServers

                    Wed Jun 5 14:46:29 2019:        maxDuration = 7200

                    Wed Jun 5 14:46:29 2019: Running Weathervane with Single-Run RunManager using Full-Run RunProcedure.

                    Wed Jun 5 14:46:29 2019: Single-Run RunManager starting run.

                    Wed Jun 5 14:46:30 2019: Stopping running services and cleaning up old log and stats files.

                    Wed Jun 5 14:46:48 2019: Preparing data for use in current run.

                    Wed Jun 5 14:46:48 2019: Configuring and starting data services for appInstance 1 of workload 1.

                    Wed Jun 5 14:47:40 2019: Backup is not available at the proper scale for 1000 users for appInstance 1 of workload 1. Loading data.

                    Wed Jun 5 14:48:29 2019: Workload 1, appInstance 1: Loading data for a maximum of 1000 users

                    Wed Jun 5 14:49:00 2019: Can't find dbloader pid for workload 1, appInstance 1

                    Wed Jun 5 14:49:29 2019: Could not properly load or prepare data for run 13.  Exiting.

                     

                    Any idea of what is going wrong?

                    • 7. Re: VMmark/Auction fails with RC=1
                      RebeccaG Expert
                      VMware Employees

                      Hi mathieugont, I'm working on this and will get you an update tomorrow.

                      • 8. Re: VMmark/Auction fails with RC=1
                        mathieugont Novice

                        Thanks a lot

                        Take your time: Monday is off here ;-)

                        • 9. Re: VMmark/Auction fails with RC=1
                          RebeccaG Expert
                          VMware Employees

                          Hi, I noticed you are using a non-default name for VMmark3.properties. Could you try reverting your file VMmark3.tile0.weathervane-auction.properties back to the default name VMmark3.properties (and following the instructions in the User's Guide to support that under "Starting a VMmark Run Through The GUI"). Also please comment out the line DS3DB/DS3DBsize = 100.

                           

                          Also, could you please zip the results file from your most recent run and attach it to this thread.


                          Thank you,

                          Rebecca