Provision fails with "unable to create VM"

[IJVidal]

Hello,

I'm looking for additional documentation regarding some issues I'm facing during the initial deployment of the virtual machines used in VMmark 3.1 environment (just building "Tile 0")

I'm struggling with various "ERROR" messages during provision (see below, please), and cannot determine the cause.

Could somebody point where to find required configurations for each VM? (besides what User's Guide says)

For example, I've set debug to "3", and in one thread that seems to provision "AuctionNoSQL0" VM, I just can read the following (in the log):

2019-05-31T04:26:43.183 [pool-3-thread-16] INFO  ST216 : VMmark3Service Thread216 Starting

2019-05-31T04:26:43.184 [pool-3-thread-16] INFO  ST216 : Validating vCServer Connection : DC Datacenter_Donado840

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 :  [R] Received 'operation:provision,Tile:0,datacenter:Datacenter_Donado840,ProvisioningSourceDatastore:HDS_G400_DS02,VMTemplate:vmmark3.1-template-020419-01,VMname:AuctionNoSQL0,HostName:gc01-vmw04.metrotel.dc2,ProvisioningDatastore:HDS_G400_DS02,PingCheck:true,HostResourcePool:,HostVMfolder:,ClientResourcePool:,ClientVMfolder:'

2019-05-31T04:26:43.258 [pool-3-thread-16] INFO  ST215 : Parsing Message

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : message: operation:provision,Tile:0,datacenter:Datacenter_Donado840,ProvisioningSourceDatastore:HDS_G400_DS02,VMTemplate:vmmark3.1-template-020419-01,VMname:AuctionNoSQL0,HostName:gc01-vmw04.metrotel.dc2,ProvisioningDatastore:HDS_G400_DS02,PingCheck:true,HostResourcePool:,HostVMfolder:,ClientResourcePool:,ClientVMfolder:

2019-05-31T04:26:43.258 [pool-3-thread-16] INFO  ST215 : Skipping Optional Parameter without entry: HostResourcePool:

2019-05-31T04:26:43.258 [pool-3-thread-16] INFO  ST215 : Skipping Optional Parameter without entry: HostVMfolder:

2019-05-31T04:26:43.258 [pool-3-thread-16] INFO  ST215 : Skipping Optional Parameter without entry: ClientResourcePool:

2019-05-31T04:26:43.258 [pool-3-thread-16] INFO  ST215 : Skipping Optional Parameter without entry: ClientVMfolder:

2019-05-31T04:26:43.258 [pool-3-thread-16] INFO  ST215 : Starting Provision Operation

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'ProvisioningSourceDatastore' : 'HDS_G400_DS02'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'ProvisioningDatastore' : 'HDS_G400_DS02'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'VMTemplate' : 'vmmark3.1-template-020419-01'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'operation' : 'provision'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'datacenter' : 'Datacenter_Donado840'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'VMname' : 'AuctionNoSQL0'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'HostName' : 'gc01-vmw04.metrotel.dc2'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'PingCheck' : 'true'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'Tile' : '0'

2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : Key Variables Populated

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : IDE 0 null 200 null IDE 0

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : IDE 1 null 201 null IDE 1

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : PS2 controller 0 null 300 null PS2 controller 0

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : PCI controller 0 null 100 null PCI controller 0

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : SIO controller 0 null 400 null SIO controller 0

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Keyboard  300 600 null Keyboard

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Pointing device 300 700 null Pointing device; Device

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Video card  100 500 null Video card

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : VMCI device 100 12000 null Device on the virtual machine PCI bus that provides support for the virtual machine communication interface

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : SCSI controller 0 100 1000 null VMware paravirtual SCSI

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : CD/DVD drive 1 201 3002 null Remote device

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Hard disk 1 1000 2000 null 16,777,216 KB

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Floppy drive 1 400 8000 null Remote

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Network adapter 1 100 4000 null DVSwitch: 50 21 40 0b 6d c4 03 77-48 b7 33 2d e2 d4 2b 8b

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Hard disk 2 1000 2001 null 209,715,200 KB

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : PVSCSI Key = 1000

2019-05-31T04:26:43.486 [pool-3-thread-16] INFO  ST215 : AuctionNoSQL0 : 192.168.39.238

2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : VMname = AuctionNoSQL0 true

2019-05-31T04:26:43.486 [pool-3-thread-16] INFO  ST215 : AuctionNoSQL0 Adding 100GB Disk to Provision

2019-05-31T04:26:43.502 [pool-3-thread-16] INFO  ST215 : dsName HDS_G400_DS02 cKey 1000 diskSizeKB 104857600 diskMode persistent dKey 1 unitNumber 1

2019-05-31T04:26:43.502 [pool-3-thread-16] INFO  ST215 : Starting Provision AuctionNoSQL0 onto Host gc01-vmw04.metrotel.dc2

2019-05-31T04:27:34.327 [pool-3-thread-16] ERROR ST215 : AuctionNoSQL0, Unable to Create VM : Msg Invalid configuration for device '0'.

2019-05-31T04:27:34.327 [pool-3-thread-16] ERROR ST215 : Terminating Early Due to Provision Thread Errors (1)

2019-05-31T04:27:34.334 [pool-3-thread-16] ERROR MAIN : Exception Caught: Terminating Early Due to Provision ThreadErrorsErrorsFound: 1 : exitSetting true

2019-05-31T04:27:34.335 [pool-3-thread-16] INFO  MAIN : Email Disabled

2019-05-31T04:27:34.335 [pool-3-thread-16] INFO  MAIN : Ending Early

But cannot find a place for checking "device 0 configuration".

Will appreciate any comment very much.

Best regards,

Ignacio

[dmorse]

Hi IJVidal

There is another thread where a user encountered this error:

ERROR ST217 : AuctionDB0, Unable to Create VM : Msg Invalid configuration for device '0'

The issue in that case appeared to be the network configuration. Could you verify that the IP address below is available?

2019-05-31T04:26:43.486 [pool-3-thread-16] INFO  ST215 : AuctionNoSQL0 : 192.168.39.238

If this is not your problem, we'll need more information about the failed provisioning.  Please run these from the PrimeClient VM:

1. cd /root/VMmark3/

2. zip -r provisioning-output.zip VMmark3.properties provisioning-output/

Then attach the resulting provisioning-output.zip as a reply to this thread.

Thanks, David

[IJVidal]

Hello DMorse!

Yes, I'd seen that thread and, in our case, it has nothing to do with used/unused IP addresses.

Sorry, I tryed to find something else reading notes and scripts under /root/VMmark3, with no success.

You'll find the zip file in the attach.

Thank you very much.

Best regards

Ignacio

[dmorse]

Ignacio,

Here is one problem:

Your VMmark3.properties states PrimeClient = 192.168.39.231, but a ProvisioningIPstaticStart = 192.168.39.221.  This means the benchmark will quickly create a VM that has an IP address

conflict with your PrimeClient -- see an example of this in your Provisioning Service logs below, where ElasticDB0 IP has been assigned an IP equal to your PrimeClient:

2019-05-31T09:13:12.398 [main] INFO  COMM : BuildingStaticIps with Static Start: 192.168.39.221 and 1 Tiles

2019-05-31T09:13:12.399 [main] DEBUG COMM : oct1 192 : oct2 168 : oct3 39 : oct4 221

2019-05-31T09:13:12.399 [main] DEBUG COMM : Client0 : 192.168.39.221

2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3WebA0 : 192.168.39.222

2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3WebB0 : 192.168.39.223

2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3WebC0 : 192.168.39.224

2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3DB0 : 192.168.39.225

2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticLB0 : 192.168.39.226

2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticWebA0 : 192.168.39.227

2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticWebB0 : 192.168.39.228

2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticAppA0 : 192.168.39.229

2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticAppB0 : 192.168.39.230

2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticDB0 : 192.168.39.231

2019-05-31T09:13:12.399 [main] DEBUG COMM : AuctionLB0 : 192.168.39.232

2019-05-31T09:13:12.399 [main] DEBUG COMM : AuctionMSQ0 : 192.168.39.233

2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionWebA0 : 192.168.39.234

2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionWebB0 : 192.168.39.235

2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionAppA0 : 192.168.39.236

2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionAppB0 : 192.168.39.237

2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionNoSQL0 : 192.168.39.238

2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionDB0 : 192.168.39.239

2019-05-31T09:13:12.400 [main] DEBUG COMM : Standby0 : 192.168.39.240

So, I would recommend changing your PrimeClient to a value well out of the reach of that range (not 192.168.39.1, since that's your gateway, but maybe 192.168.39.100, .150, or some other free IP).

The other thing I'd like you to check is vCenter Server as you're trying to create the first tile -- specifically, look in the logs/events for any errors that occur while these VMs are being created.  For example, maybe it can't create a VM because of insufficient disk space on one of your datastores.

Hope this helps,

David

[IJVidal]

Hello David,

Yes: there is an issue with that IP address, and I changed it as suggested.

But, after retrying, I saw the same error message.

Reviewing task logs under vcenter I found that there's an error during clone operation of the initial template ("vmmark3.1-template-020419-01"), just with the same message ("Invalid configuration for device '0'").

Besides that, I also see that (nearly) all VM's were created, but remain powered off:

If I try to power on one of them, I get other message, and I believe here is the cause of the message I fist saw during provisioning.

Here´s the events sequence for one of the VM's (it is: DS3WebB0):

I see there's an indication about "could not find the file".

At the "vmx" file for this VM, there's a configuration line for:

nvram = "ovf:/file/file2"

You can see the file DS3WebB0 in the attach.

Is it possible to comment out this line?

On the other hand... I think these are different problems (one related with cloning the template, and other with this line "nvram=ovf:/file/file2" at vmx file).

Will keep trying to do some additional research.

Thank you very much.

Kind regards

Ignacio

[dmorse]

A few things:

• You should be using separate clusters for the VMmark3Cluster and the ClientCluster; currently they are the same:

// String vCServerVMmark3Cluster : Default empty : Ex vCServerVMmark3Cluster = Cluster1

vCServerVMmark3Cluster = Cluster_UCS01

// String vCServerClientCluster : Default empty : Ex vCServerClientCluster = ClientCluster1 : This is the cluster of hosts running your

//    Client VMs and is where provisioning will place new client VMs.

vCServerClientCluster = Cluster_UCS01

• From your screenshot, it looks like you're creating resource pools, which are not necessary (and I would not recommend).

• What builds of ESXi and vCenter Server are you using (i.e. 6.5 or 6.7?  exact build #s?)

• Could you try to manually clone the vmmark3.1-template-020419 VM (you can give it any name, such as test-vmmark-vm), put it on the DPG_vSPHERE_INFRA network, power it on, and see if it does?

[IJVidal]

David,

OK, I see...

Need some time for cluster reorganization (will use 2 new hosts, should connect them with vCenter).

Configured resource pools have all unlimited resources. One of the aspects we wanted to check was setting limits to resource pools (after testing), but I understand that for benchmarking it'll be more clear not to use them.

Will correct this point too.

Regarding software versions, we're using...

- For ESXi: VMware-ESXi-6.5.0-9298722-Custom-Cisco-6.5.2.2 (from MyVMWare's site, just "unboxed"... no patches, no special settings, it runs on Cisco UCS hardware)

- For vCenter: Versión 6.5.0.23000 Build 11347054

And, at last, the exercise of manually cloning the template was OK (ended with no errors).

But once you try to power on the VM ("CLONE_T"), you get this sequence of events:

Will follow up later (maybe this night), I have a meeting starting soon.

Thank you very much for your insights.

Kind regards

Ignacio

[dmorse]

Thank you for trying the clone and power on.  The fact that it failed in the same way (the ovf file2), tells me it may be a problem with the OVA file, at least in a 6.5 environment.

I am checking with our internal VMmark team to see if it's related to this issue:

6.5 fails to boot OVA imported VM because of bogus nvram file reference

If we generated this OVA file with 6.7, that could be why you're seeing this error in a 6.5 environment.  In that case, trying to remove the nvram line (as you mentioned before) may be required, but that seems far from ideal.  The better option may be to upgrade to 6.7, if that is an option?

[IJVidal]

Hello David,

Sorry for the delay.

I believe we are going to upgrade to 6.7 during this year (Q4), but for the next months 6.5 will be there...

Thinking about what you said, and as we wanted to test how this new installation could work running some stressed workloads (as a benchmark, and for our records), I believe that other alternative could be running an older version of VMmark (adequate for 6.5).

So, on Monday I'll check that.

Thank you very much for your follow-up.

Kind regards,

  Ignacio

[dmorse]

While using the previous VMmark release (3.0) is an option and would likely work, I spoke to the developer who created the OVA template, and he was able to reproduce your issue with a 6.5 vCenter and 6.5 ESXi host when deploying the OVA template through the vSphere Client.

However, he said deploying the template via the latest PowerCLI instead does work.  For steps on how to do this, see the "Problems Deploying the VMmark Template" in the User's Guide.  It involves installing PowerShell and VMware PowerCLI, then deploying the template to a host via the Import-VApp cmdlet.

[IJVidal]

Ok, I see.

So, I'll try using PowerCLI and VMmark 3.1 and send some feedback about how it worked.

Please, give me some time for testing.

Appreciate very much your suggestions.

Kind regards

Ignacio