only “Client0” VM was deployed

[ahashimoto]

Hi,

I started the provisioning process, from the command line execute:

# java -jar tools/VMmark3Service.jar -c VMmark3.properties

Looks the provisioning process completed successfully but only “Client0” VM was deployed. There are no any other VMs lile AuctionWeb, DS3DB0, etc in the cluster for some reason.

Do you have any idea the reason why other VMs could not be deployed?

[fredab2]

I looked through the uploaded data and VMmark3.properties file and don't understand why the provisioning stopped unless there was a special charater in the VMmark3.properties file.

Can you try the following.

1. Delete the Client0 VM that was created.

2. Edit the following lines (287 and 288) in your VMmark3.properties file as shown below so DS4WebA comes before Client:

285 //   Wrapped Format Example:  (Note an example is provided below, users should replace the example LUN values prior to provisioning)

286 ProvisioningDatastores = \

287 DS3WebA:vsanDatastore,\

288 Client:vsanDatastore,\

3. try to do another provisioning with the jar command.

Thanks and reply with the results.

Fred

[ahashimoto]

Thank you for your advice.

I just tried the following procedure and I saw that only DS3WebA0 was created instead of Client0 but there were still nothing else.

i realized some difference in the log between this environment and other environment where I did deploy before.

1.

It needed much more time to go ahead provisioning after "3 Hosts Under Test" compared to other environment. It took about 1 minutes.

2020-01-29T11:02:20.745 [main] INFO  vCSc : 3 Active VMmark3 Hosts

2020-01-29T11:02:20.745 [main] INFO  vCSc :   3 Hosts Under Test

2020-01-29T11:03:12.687 [main] INFO  vCSc : Provisioning Source: vmmark3.1-template-020419 : centos64guest Found

2.

I could see the following at this time. I know there were not any VMs in provisioning at first so I'm not sure I could see such warnings at this time only.

2020-01-29T11:03:16.569 [main] WARN  MAIN : Existing VM Client0 NOT FOUND

2020-01-29T11:03:16.590 [main] WARN  MAIN : Existing VM DS3WebB0 NOT FOUND

2020-01-29T11:03:16.611 [main] WARN  MAIN : Existing VM DS3WebC0 NOT FOUND

:

2020-01-29T11:03:17.045 [main] WARN  MAIN : Existing VM Standby0 NOT FOUND

3.

And just one thread in the pool was invoked at this time. I assume that many threads should be invoked in order to create VMs.

2020-01-29T11:03:17.360 [pool-3-thread-1] ...

Thanks

[fredab2]

It appears the provisioning stops after the line .

Client:vsanDatastore,\

I am guessing there is a special character in that line before the line feed.

Try changing the VMmark3.properties file to 1 single line below:

ProvisioningDatastores = Client:vsanDatastore,DS3WebA:vsanDatastore,DS3WebB:vsanDatastore,DS3WebC:vsanDatastore,DS3DB:vsanDatastore,AuctionLB:vsanDatastore,AuctionMSQ:vsanDatastore,AuctionWebA:vsanDatastore,AuctionWebB:vsanDatastore,AuctionAppA:vsanDatastore,AuctionAppB:vsanDatastore,AuctionNoSQL:vsanDatastore,AuctionDB:vsanDatastore,ElasticLB:vsanDatastore,ElasticWebA:vsanDatastore,ElasticWebB:vsanDatastore,ElasticAppA:vsanDatastore,ElasticAppB:vsanDatastore,ElasticDB:vsanDatastore,Standby:vsanDatastore

then comment out or remove all the wrapped format lines below:

288 ProvisioningDatastores = \

289 DS3WebA:vsanDatastore,\

290 Client:vsanDatastore,\

291 DS3WebB:vsanDatastore,\

292 DS3WebC:vsanDatastore,\

293 DS3DB:vsanDatastore,\

294 AuctionLB:vsanDatastore,\

295 AuctionMSQ:vsanDatastore,\

296 AuctionWebA:vsanDatastore,\

297 AuctionWebB:vsanDatastore,\

298 AuctionAppA:vsanDatastore,\

299 AuctionAppB:vsanDatastore,\

300 AuctionNoSQL:vsanDatastore,\

301 AuctionDB:vsanDatastore,\

302 ElasticLB:vsanDatastore,\

303 ElasticWebA:vsanDatastore,\

304 ElasticWebB:vsanDatastore,\

305 ElasticAppA:vsanDatastore,\

306 ElasticAppB:vsanDatastore,\

307 ElasticDB:vsanDatastore,\

308 Standby:vsanDatastore

[ahashimoto]

Thanks. that was correct. After changing ProvisioningDatastore to 1 single line, VMmark started to provision each VM. Probably, this issue was resolved.

However, another issue appeared. Even though Vmmark continued to provision VMs but exception errors appeared intermittently (attached) for some thread-#. I suspect if this result from a kind of timeout and I wonder if it might be resource or performance issue.  Have you experienced this error?

[fredab2]

Can you attach and send the provisioning directory from this attempted provisioning run?

Fred

[ahashimoto]

Here is the provisioning files as attached. This provisioning started yesterday and looks still on going to build DS3DB0 (about over 20 hours now). I'm not sure the provisioning process was recovered and continues correctly after the error messages i sent you yesterday. But it maybe fine after that actually so I'm just waiting for finishing DS3DB build process.

[fredab2]

Looking at the  VMmark3-ProvisioningService.log you uploaded it looks like it finished without errors and you are just waiting on the DS3DB0 DB to finish.

Creation Status: ssh DS3DB0 'tail -f /ds3/VMmark3-DS3installoutput.txt'

Error Output:: Found 0 Errors

TimeElapsed: 3207 (secs)

2020-01-29T17:46:32.665 [main] INFO  MAIN : Stopping STAX if present

2020-01-29T17:46:32.665 [main] DEBUG MAIN : Stax stop cmd: perl /root/VMmark3/tools/VMmark3ServiceComm.pl restartstaf

2020-01-29T17:46:32.849 [main] DEBUG MAIN : StafRestart Ouput: perl /root/VMmark3/tools/VMmark3ServiceComm.pl restartstaf ::

/root/VMmark3/tools/VMmark3ServiceComm.pl : v.4

----------------------------------------------------------------------------

2020-01-29T17:46:32.849 [main] INFO  MAIN : Restarting PrimeClient Staf...

2020-01-29T17:46:32.849 [main] DEBUG MAIN : StafRestart Cmd: systemctl restart startstaf.service

2020-01-29T17:46:34.076 [main] DEBUG MAIN : StafRestart Ouput: systemctl restart startstaf.service ::

----------------------------------------------------------------------------

2020-01-29T17:46:34.076 [main] INFO  MAIN : Users should start STAX prior to beginning benchmark runs : See the desktop icon

2020-01-29T17:46:34.076 [main] INFO  MAIN : Successfully restarted prime client staf

2020-01-29T17:46:34.076 [main] INFO  MAIN : Final: VMmark3Service_v1.0.17 Finished

Summary::

/root/VMmark3/provisioning-output/vmmark3-output/

Automation Ending:

Please add the contents of hosts-stub.txt to your PrimeClient's /etc/hosts file

STAX should be started before beginning new runs

DS3DB0 is still being created : Review VM for status::  :: Detail (if applicable): IP: 10.0.12.107

Creation Status: ssh DS3DB0 'tail -f /ds3/VMmark3-DS3installoutput.txt'

Error Output:: Found 0 Errors

TimeElapsed: 3207 (secs)

Error Output:: Found 0 Errors

TimeElapsed: 3208 (secs)

2020-01-29T17:46:34.076 [main] INFO  MAIN : VMmark3Service_v1.0.17 Exiting Normally

[ahashimoto]

Great. Thanks..Unfortunately, it ended up with DB build fail because of lack of space even though it had more than 1TB. I will try to provision again after expanding the space.