Hello,
I'm looking for additional documentation regarding some issues I'm facing during the initial deployment of the virtual machines used in VMmark 3.1 environment (just building "Tile 0")
I'm struggling with various "ERROR" messages during provision (see below, please), and cannot determine the cause.
Could somebody point where to find required configurations for each VM? (besides what User's Guide says)
For example, I've set debug to "3", and in one thread that seems to provision "AuctionNoSQL0" VM, I just can read the following (in the log):
2019-05-31T04:26:43.183 [pool-3-thread-16] INFO ST216 : VMmark3Service Thread216 Starting
2019-05-31T04:26:43.184 [pool-3-thread-16] INFO ST216 : Validating vCServer Connection : DC Datacenter_Donado840
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : [R] Received 'operation:provision,Tile:0,datacenter:Datacenter_Donado840,ProvisioningSourceDatastore:HDS_G400_DS02,VMTemplate:vmmark3.1-template-020419-01,VMname:AuctionNoSQL0,HostName:gc01-vmw04.metrotel.dc2,ProvisioningDatastore:HDS_G400_DS02,PingCheck:true,HostResourcePool:,HostVMfolder:,ClientResourcePool:,ClientVMfolder:'
2019-05-31T04:26:43.258 [pool-3-thread-16] INFO ST215 : Parsing Message
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : message: operation:provision,Tile:0,datacenter:Datacenter_Donado840,ProvisioningSourceDatastore:HDS_G400_DS02,VMTemplate:vmmark3.1-template-020419-01,VMname:AuctionNoSQL0,HostName:gc01-vmw04.metrotel.dc2,ProvisioningDatastore:HDS_G400_DS02,PingCheck:true,HostResourcePool:,HostVMfolder:,ClientResourcePool:,ClientVMfolder:
2019-05-31T04:26:43.258 [pool-3-thread-16] INFO ST215 : Skipping Optional Parameter without entry: HostResourcePool:
2019-05-31T04:26:43.258 [pool-3-thread-16] INFO ST215 : Skipping Optional Parameter without entry: HostVMfolder:
2019-05-31T04:26:43.258 [pool-3-thread-16] INFO ST215 : Skipping Optional Parameter without entry: ClientResourcePool:
2019-05-31T04:26:43.258 [pool-3-thread-16] INFO ST215 : Skipping Optional Parameter without entry: ClientVMfolder:
2019-05-31T04:26:43.258 [pool-3-thread-16] INFO ST215 : Starting Provision Operation
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'ProvisioningSourceDatastore' : 'HDS_G400_DS02'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'ProvisioningDatastore' : 'HDS_G400_DS02'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'VMTemplate' : 'vmmark3.1-template-020419-01'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'operation' : 'provision'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'datacenter' : 'Datacenter_Donado840'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'VMname' : 'AuctionNoSQL0'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'HostName' : 'gc01-vmw04.metrotel.dc2'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'PingCheck' : 'true'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : RMQ Thread Message 'Tile' : '0'
2019-05-31T04:26:43.258 [pool-3-thread-16] DEBUG ST215 : Key Variables Populated
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : IDE 0 null 200 null IDE 0
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : IDE 1 null 201 null IDE 1
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : PS2 controller 0 null 300 null PS2 controller 0
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : PCI controller 0 null 100 null PCI controller 0
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : SIO controller 0 null 400 null SIO controller 0
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Keyboard 300 600 null Keyboard
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Pointing device 300 700 null Pointing device; Device
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Video card 100 500 null Video card
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : VMCI device 100 12000 null Device on the virtual machine PCI bus that provides support for the virtual machine communication interface
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : SCSI controller 0 100 1000 null VMware paravirtual SCSI
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : CD/DVD drive 1 201 3002 null Remote device
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Hard disk 1 1000 2000 null 16,777,216 KB
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Floppy drive 1 400 8000 null Remote
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Network adapter 1 100 4000 null DVSwitch: 50 21 40 0b 6d c4 03 77-48 b7 33 2d e2 d4 2b 8b
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : Device : Hard disk 2 1000 2001 null 209,715,200 KB
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : PVSCSI Key = 1000
2019-05-31T04:26:43.486 [pool-3-thread-16] INFO ST215 : AuctionNoSQL0 : 192.168.39.238
2019-05-31T04:26:43.486 [pool-3-thread-16] DEBUG ST215 : VMname = AuctionNoSQL0 true
2019-05-31T04:26:43.486 [pool-3-thread-16] INFO ST215 : AuctionNoSQL0 Adding 100GB Disk to Provision
2019-05-31T04:26:43.502 [pool-3-thread-16] INFO ST215 : dsName HDS_G400_DS02 cKey 1000 diskSizeKB 104857600 diskMode persistent dKey 1 unitNumber 1
2019-05-31T04:26:43.502 [pool-3-thread-16] INFO ST215 : Starting Provision AuctionNoSQL0 onto Host gc01-vmw04.metrotel.dc2
2019-05-31T04:27:34.327 [pool-3-thread-16] ERROR ST215 : AuctionNoSQL0, Unable to Create VM : Msg Invalid configuration for device '0'.
2019-05-31T04:27:34.327 [pool-3-thread-16] ERROR ST215 : Terminating Early Due to Provision Thread Errors (1)
2019-05-31T04:27:34.334 [pool-3-thread-16] ERROR MAIN : Exception Caught: Terminating Early Due to Provision ThreadErrorsErrorsFound: 1 : exitSetting true
2019-05-31T04:27:34.335 [pool-3-thread-16] INFO MAIN : Email Disabled
2019-05-31T04:27:34.335 [pool-3-thread-16] INFO MAIN : Ending Early
But cannot find a place for checking "device 0 configuration".
Will appreciate any comment very much.
Best regards,
Ignacio
Hi IJVidal
There is another thread where a user encountered this error:
ERROR ST217 : AuctionDB0, Unable to Create VM : Msg Invalid configuration for device '0'
The issue in that case appeared to be the network configuration. Could you verify that the IP address below is available?
2019-05-31T04:26:43.486 [pool-3-thread-16] INFO ST215 : AuctionNoSQL0 : 192.168.39.238
If this is not your problem, we'll need more information about the failed provisioning. Please run these from the PrimeClient VM:
cd /root/VMmark3/
zip -r provisioning-output.zip VMmark3.properties provisioning-output/
Then attach the resulting provisioning-output.zip as a reply to this thread.
Thanks, David
Hello DMorse!
Yes, I'd seen that thread and, in our case, it has nothing to do with used/unused IP addresses.
Sorry, I tryed to find something else reading notes and scripts under /root/VMmark3, with no success.
You'll find the zip file in the attach.
Thank you very much.
Best regards
Ignacio
Ignacio,
Here is one problem:
Your VMmark3.properties states PrimeClient = 192.168.39.231, but a ProvisioningIPstaticStart = 192.168.39.221. This means the benchmark will quickly create a VM that has an IP address
conflict with your PrimeClient -- see an example of this in your Provisioning Service logs below, where ElasticDB0 IP has been assigned an IP equal to your PrimeClient:
2019-05-31T09:13:12.398 [main] INFO COMM : BuildingStaticIps with Static Start: 192.168.39.221 and 1 Tiles
2019-05-31T09:13:12.399 [main] DEBUG COMM : oct1 192 : oct2 168 : oct3 39 : oct4 221
2019-05-31T09:13:12.399 [main] DEBUG COMM : Client0 : 192.168.39.221
2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3WebA0 : 192.168.39.222
2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3WebB0 : 192.168.39.223
2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3WebC0 : 192.168.39.224
2019-05-31T09:13:12.399 [main] DEBUG COMM : DS3DB0 : 192.168.39.225
2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticLB0 : 192.168.39.226
2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticWebA0 : 192.168.39.227
2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticWebB0 : 192.168.39.228
2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticAppA0 : 192.168.39.229
2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticAppB0 : 192.168.39.230
2019-05-31T09:13:12.399 [main] DEBUG COMM : ElasticDB0 : 192.168.39.231
2019-05-31T09:13:12.399 [main] DEBUG COMM : AuctionLB0 : 192.168.39.232
2019-05-31T09:13:12.399 [main] DEBUG COMM : AuctionMSQ0 : 192.168.39.233
2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionWebA0 : 192.168.39.234
2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionWebB0 : 192.168.39.235
2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionAppA0 : 192.168.39.236
2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionAppB0 : 192.168.39.237
2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionNoSQL0 : 192.168.39.238
2019-05-31T09:13:12.400 [main] DEBUG COMM : AuctionDB0 : 192.168.39.239
2019-05-31T09:13:12.400 [main] DEBUG COMM : Standby0 : 192.168.39.240
So, I would recommend changing your PrimeClient to a value well out of the reach of that range (not 192.168.39.1, since that's your gateway, but maybe 192.168.39.100, .150, or some other free IP).
The other thing I'd like you to check is vCenter Server as you're trying to create the first tile -- specifically, look in the logs/events for any errors that occur while these VMs are being created. For example, maybe it can't create a VM because of insufficient disk space on one of your datastores.
Hope this helps,
David
Hello David,
Yes: there is an issue with that IP address, and I changed it as suggested.
But, after retrying, I saw the same error message.
Reviewing task logs under vcenter I found that there's an error during clone operation of the initial template ("vmmark3.1-template-020419-01"), just with the same message ("Invalid configuration for device '0'").
Besides that, I also see that (nearly) all VM's were created, but remain powered off:
If I try to power on one of them, I get other message, and I believe here is the cause of the message I fist saw during provisioning.
Here´s the events sequence for one of the VM's (it is: DS3WebB0):
I see there's an indication about "could not find the file".
At the "vmx" file for this VM, there's a configuration line for:
nvram = "ovf:/file/file2"
You can see the file DS3WebB0 in the attach.
Is it possible to comment out this line?
On the other hand... I think these are different problems (one related with cloning the template, and other with this line "nvram=ovf:/file/file2" at vmx file).
Will keep trying to do some additional research.
Thank you very much.
Kind regards
Ignacio
A few things:
// String vCServerVMmark3Cluster : Default empty : Ex vCServerVMmark3Cluster = Cluster1
vCServerVMmark3Cluster = Cluster_UCS01
// String vCServerClientCluster : Default empty : Ex vCServerClientCluster = ClientCluster1 : This is the cluster of hosts running your
// Client VMs and is where provisioning will place new client VMs.
vCServerClientCluster = Cluster_UCS01
David,
OK, I see...
Need some time for cluster reorganization (will use 2 new hosts, should connect them with vCenter).
Configured resource pools have all unlimited resources. One of the aspects we wanted to check was setting limits to resource pools (after testing), but I understand that for benchmarking it'll be more clear not to use them.
Will correct this point too.
Regarding software versions, we're using...
- For ESXi: VMware-ESXi-6.5.0-9298722-Custom-Cisco-6.5.2.2 (from MyVMWare's site, just "unboxed"... no patches, no special settings, it runs on Cisco UCS hardware)
- For vCenter: Versión 6.5.0.23000 Build 11347054
And, at last, the exercise of manually cloning the template was OK (ended with no errors).
But once you try to power on the VM ("CLONE_T"), you get this sequence of events:
Will follow up later (maybe this night), I have a meeting starting soon.
Thank you very much for your insights.
Kind regards
Ignacio
Thank you for trying the clone and power on. The fact that it failed in the same way (the ovf file2), tells me it may be a problem with the OVA file, at least in a 6.5 environment.
I am checking with our internal VMmark team to see if it's related to this issue:
6.5 fails to boot OVA imported VM because of bogus nvram file reference
If we generated this OVA file with 6.7, that could be why you're seeing this error in a 6.5 environment. In that case, trying to remove the nvram line (as you mentioned before) may be required, but that seems far from ideal. The better option may be to upgrade to 6.7, if that is an option?
Hello David,
Sorry for the delay.
I believe we are going to upgrade to 6.7 during this year (Q4), but for the next months 6.5 will be there...
Thinking about what you said, and as we wanted to test how this new installation could work running some stressed workloads (as a benchmark, and for our records), I believe that other alternative could be running an older version of VMmark (adequate for 6.5).
So, on Monday I'll check that.
Thank you very much for your follow-up.
Kind regards,
Ignacio
While using the previous VMmark release (3.0) is an option and would likely work, I spoke to the developer who created the OVA template, and he was able to reproduce your issue with a 6.5 vCenter and 6.5 ESXi host when deploying the OVA template through the vSphere Client.
However, he said deploying the template via the latest PowerCLI instead does work. For steps on how to do this, see the "Problems Deploying the VMmark Template" in the User's Guide. It involves installing PowerShell and VMware PowerCLI, then deploying the template to a host via the Import-VApp cmdlet.
Ok, I see.
So, I'll try using PowerCLI and VMmark 3.1 and send some feedback about how it worked.
Please, give me some time for testing.
Appreciate very much your suggestions.
Kind regards
Ignacio