8 Replies Latest reply on Jan 31, 2020 8:31 PM by ahashimoto

    only “Client0” VM was deployed

    ahashimoto Lurker
    VMware Employees


      I started the provisioning process, from the command line execute:

      # java -jar tools/VMmark3Service.jar -c VMmark3.properties

      Looks the provisioning process completed successfully but only “Client0” VM was deployed. There are no any other VMs lile AuctionWeb, DS3DB0, etc in the cluster for some reason.

      Do you have any idea the reason why other VMs could not be deployed?


        • 1. Re: only “Client0” VM was deployed
          fredab2 Novice
          VMware Employees

          I looked through the uploaded data and VMmark3.properties file and don't understand why the provisioning stopped unless there was a special charater in the VMmark3.properties file.


          Can you try the following.


          1. Delete the Client0 VM that was created.

          2. Edit the following lines (287 and 288) in your VMmark3.properties file as shown below so DS4WebA comes before Client:


          285 //   Wrapped Format Example:  (Note an example is provided below, users should replace the example LUN values prior to provisioning)

          286 ProvisioningDatastores = \

          287 DS3WebA:vsanDatastore,\

          288 Client:vsanDatastore,\


          3. try to do another provisioning with the jar command.


          Thanks and reply with the results.



          • 2. Re: only “Client0” VM was deployed
            ahashimoto Lurker
            VMware Employees

            Thank you for your advice.

            I just tried the following procedure and I saw that only DS3WebA0 was created instead of Client0 but there were still nothing else.


            i realized some difference in the log between this environment and other environment where I did deploy before.



            It needed much more time to go ahead provisioning after "3 Hosts Under Test" compared to other environment. It took about 1 minutes.

            2020-01-29T11:02:20.745 [main] INFO  vCSc : 3 Active VMmark3 Hosts

            2020-01-29T11:02:20.745 [main] INFO  vCSc :   3 Hosts Under Test

            2020-01-29T11:03:12.687 [main] INFO  vCSc : Provisioning Source: vmmark3.1-template-020419 : centos64guest Found



            I could see the following at this time. I know there were not any VMs in provisioning at first so I'm not sure I could see such warnings at this time only.

            2020-01-29T11:03:16.569 [main] WARN  MAIN : Existing VM Client0 NOT FOUND

            2020-01-29T11:03:16.590 [main] WARN  MAIN : Existing VM DS3WebB0 NOT FOUND

            2020-01-29T11:03:16.611 [main] WARN  MAIN : Existing VM DS3WebC0 NOT FOUND


            2020-01-29T11:03:17.045 [main] WARN  MAIN : Existing VM Standby0 NOT FOUND



            And just one thread in the pool was invoked at this time. I assume that many threads should be invoked in order to create VMs.

            2020-01-29T11:03:17.360 [pool-3-thread-1] ...



            • 3. Re: only “Client0” VM was deployed
              fredab2 Novice
              VMware Employees

              It appears the provisioning stops after the line .




              I am guessing there is a special character in that line before the line feed.


              Try changing the VMmark3.properties file to 1 single line below:


              ProvisioningDatastores = Client:vsanDatastore,DS3WebA:vsanDatastore,DS3WebB:vsanDatastore,DS3WebC:vsanDatastore,DS3DB:vsanDatastore,AuctionLB:vsanDatastore,AuctionMSQ:vsanDatastore,AuctionWebA:vsanDatastore,AuctionWebB:vsanDatastore,AuctionAppA:vsanDatastore,AuctionAppB:vsanDatastore,AuctionNoSQL:vsanDatastore,AuctionDB:vsanDatastore,ElasticLB:vsanDatastore,ElasticWebA:vsanDatastore,ElasticWebB:vsanDatastore,ElasticAppA:vsanDatastore,ElasticAppB:vsanDatastore,ElasticDB:vsanDatastore,Standby:vsanDatastore


              then comment out or remove all the wrapped format lines below:


              288 ProvisioningDatastores = \

              289 DS3WebA:vsanDatastore,\

              290 Client:vsanDatastore,\

              291 DS3WebB:vsanDatastore,\

              292 DS3WebC:vsanDatastore,\

              293 DS3DB:vsanDatastore,\

              294 AuctionLB:vsanDatastore,\

              295 AuctionMSQ:vsanDatastore,\

              296 AuctionWebA:vsanDatastore,\

              297 AuctionWebB:vsanDatastore,\

              298 AuctionAppA:vsanDatastore,\

              299 AuctionAppB:vsanDatastore,\

              300 AuctionNoSQL:vsanDatastore,\

              301 AuctionDB:vsanDatastore,\

              302 ElasticLB:vsanDatastore,\

              303 ElasticWebA:vsanDatastore,\

              304 ElasticWebB:vsanDatastore,\

              305 ElasticAppA:vsanDatastore,\

              306 ElasticAppB:vsanDatastore,\

              307 ElasticDB:vsanDatastore,\

              308 Standby:vsanDatastore

              • 4. Re: only “Client0” VM was deployed
                ahashimoto Lurker
                VMware Employees

                Thanks. that was correct. After changing ProvisioningDatastore to 1 single line, VMmark started to provision each VM. Probably, this issue was resolved.

                However, another issue appeared. Even though Vmmark continued to provision VMs but exception errors appeared intermittently (attached) for some thread-#. I suspect if this result from a kind of timeout and I wonder if it might be resource or performance issue.  Have you experienced this error?

                • 5. Re: only “Client0” VM was deployed
                  fredab2 Novice
                  VMware Employees

                  Can you attach and send the provisioning directory from this attempted provisioning run?



                  • 6. Re: only “Client0” VM was deployed
                    ahashimoto Lurker
                    VMware Employees

                    Here is the provisioning files as attached. This provisioning started yesterday and looks still on going to build DS3DB0 (about over 20 hours now). I'm not sure the provisioning process was recovered and continues correctly after the error messages i sent you yesterday. But it maybe fine after that actually so I'm just waiting for finishing DS3DB build process.

                    • 7. Re: only “Client0” VM was deployed
                      fredab2 Novice
                      VMware Employees

                      Looking at the  VMmark3-ProvisioningService.log you uploaded it looks like it finished without errors and you are just waiting on the DS3DB0 DB to finish.


                      Creation Status: ssh DS3DB0 'tail -f /ds3/VMmark3-DS3installoutput.txt'

                      Error Output:: Found 0 Errors


                      TimeElapsed: 3207 (secs)

                      2020-01-29T17:46:32.665 [main] INFO  MAIN : Stopping STAX if present

                      2020-01-29T17:46:32.665 [main] DEBUG MAIN : Stax stop cmd: perl /root/VMmark3/tools/VMmark3ServiceComm.pl restartstaf

                      2020-01-29T17:46:32.849 [main] DEBUG MAIN : StafRestart Ouput: perl /root/VMmark3/tools/VMmark3ServiceComm.pl restartstaf ::

                      /root/VMmark3/tools/VMmark3ServiceComm.pl : v.4




                      2020-01-29T17:46:32.849 [main] INFO  MAIN : Restarting PrimeClient Staf...

                      2020-01-29T17:46:32.849 [main] DEBUG MAIN : StafRestart Cmd: systemctl restart startstaf.service

                      2020-01-29T17:46:34.076 [main] DEBUG MAIN : StafRestart Ouput: systemctl restart startstaf.service ::



                      2020-01-29T17:46:34.076 [main] INFO  MAIN : Users should start STAX prior to beginning benchmark runs : See the desktop icon

                      2020-01-29T17:46:34.076 [main] INFO  MAIN : Successfully restarted prime client staf

                      2020-01-29T17:46:34.076 [main] INFO  MAIN : Final: VMmark3Service_v1.0.17 Finished




                      Automation Ending:

                      Please add the contents of hosts-stub.txt to your PrimeClient's /etc/hosts file

                      STAX should be started before beginning new runs

                      DS3DB0 is still being created : Review VM for status::  :: Detail (if applicable): IP:

                      Creation Status: ssh DS3DB0 'tail -f /ds3/VMmark3-DS3installoutput.txt'

                      Error Output:: Found 0 Errors

                      TimeElapsed: 3207 (secs)

                      Error Output:: Found 0 Errors

                      TimeElapsed: 3208 (secs)

                      2020-01-29T17:46:34.076 [main] INFO  MAIN : VMmark3Service_v1.0.17 Exiting Normally

                      • 8. Re: only “Client0” VM was deployed
                        ahashimoto Lurker
                        VMware Employees

                        Great. Thanks..Unfortunately, it ended up with DB build fail because of lack of space even though it had more than 1TB. I will try to provision again after expanding the space.