All Posts

how to fix with this problem , please  [root@primeclient VMmark3]# java -jar tools/VMmark3Service.jar -c VMmark3.properties VMmark3Service_v1.0.17 Reading VMmark3 Properties File ConfigFile = VMm... See more...
how to fix with this problem , please  [root@primeclient VMmark3]# java -jar tools/VMmark3Service.jar -c VMmark3.properties VMmark3Service_v1.0.17 Reading VMmark3 Properties File ConfigFile = VMmark3.properties 2023-06-16T18:17:29.247 [main] INFO MAIN : VMmark3Service_v1.0.17 2023-06-16T18:17:29.250 [main] INFO MAIN : ConfigFile: VMmark3.properties Loaded 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: AllowRunOverwrite: true 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: EmailNotifications: true 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: EmailList: [] 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: EmailSMTP: 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: EmailFrom: 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: EmailPort: 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: RunName: vmmark3-1tiles 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: PrimeClient: 192.168.1.251 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: vCServer: 192.168.1.250 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: vCServerUsername: administrator@vsphere.local 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: vCServerPassword: ******** 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: vCServerDatacenter: Datacenter 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: Tiles: 1 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningIPstaticStart: 192.168.1.1 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningIPmode: static 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningSource: tile0 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningNumTiles: 1 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningDatastores: Client:datastore1,DS3WebA:datastore2,DS3WebB:datastore2,DS3WebC:datastore2,DS3DB:datastore2,AuctionLB:datastore2,AuctionMSQ:datastore2,AuctionWebA:datastore2,AuctionWebB:datastore2,AuctionAppA:datastore2,AuctionAppB:datastore2,AuctionNoSQL:datastore2,AuctionDB:datastore2,ElasticLB:datastore2,ElasticWebA:datastore2,ElasticWebB:datastore2,ElasticAppA:datastore2,ElasticAppB:datastore2,ElasticDB:datastore2,Standby:datastore2 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningTimeZone: UTC 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningGateway: 192.168.1.254 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningSubnetMask: 255.255.255.0 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningNetworkLabel: VM Network 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisioningNetworkType: standard 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: ProvisionPingIpSeconds: 600 2023-06-16T18:17:29.251 [main] INFO MAIN : DeployVMinfo: DeployVM0:192.168.1.245,DeployVM1:192.168.1.246 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: vCServerVMmark3Cluster: ClientCluster 2023-06-16T18:17:29.251 [main] INFO MAIN : Config:: vCServerClientCluster: ClientCluster 2023-06-16T18:17:29.251 [main] INFO MAIN : Initializing Env 2023-06-16T18:17:29.251 [main] INFO MAIN : AllowRunOverwrite Enabled, overwriting directory 2023-06-16T18:17:29.252 [main] INFO MAIN : Hosts file Stub Output: hosts-stub.txt 2023-06-16T18:17:29.252 [main] INFO MAIN : ConfigFileName: VMmark3.properties 2023-06-16T18:17:29.252 [main] INFO MAIN : Copy File: /root/VMmark3/VMmark3.properties : /root/VMmark3/provisioning-output/vmmark3-1tiles/VMmark3.properties 2023-06-16T18:17:29.281 [main] ERROR MAIN : Email Notifications Disabled due to Email Settings Errors 2023-06-16T18:17:29.281 [main] INFO MAIN : Initializing Static VMmapping Info 2023-06-16T18:17:29.283 [main] INFO COMM : BuildingStaticIps with Static Start: 192.168.1.1 and 1 Tiles 2023-06-16T18:17:29.283 [main] INFO COMM : Working with User Defined DeployVMinfo: DeployVM0:192.168.1.245,DeployVM1:192.168.1.246 2023-06-16T18:17:29.283 [main] INFO MAIN : Initialization Completed Successfully 2023-06-16T18:17:29.283 [main] INFO MAIN : Starting VMmark3 Provisioning Service 2023-06-16T18:17:29.283 [main] INFO MAIN : Logging Into vCServer: 192.168.1.250 2023-06-16T18:17:29.503 [main] INFO MAIN : Validating Setup Parameters 2023-06-16T18:17:29.503 [main] INFO MAIN : Provisioning DS3DB is Not Active 2023-06-16T18:17:29.503 [main] INFO MAIN : Setup Completed Successfully 2023-06-16T18:17:29.513 [main] INFO vCSc : Validating Provisioning Parameters 2023-06-16T18:17:29.516 [main] INFO vCSc : vCServerDatacenter: Datacenter Found 2023-06-16T18:17:29.516 [main] INFO vCSc : vCServerDatacenter: Datacenter Found 2023-06-16T18:17:29.516 [main] INFO vCSc : Scanning Client Cluster ClientCluster for Host Systems 2023-06-16T18:17:29.552 [main] INFO vCSc : Host[0] : 192.168.1.253 : Connection connected : Power poweredOn : MaintenanceMode false : Active 2023-06-16T18:17:29.555 [main] INFO vCSc : 1 Active VMmark3 Hosts 2023-06-16T18:17:29.555 [main] INFO vCSc : 1 Client Hosts Under Test 2023-06-16T18:17:29.555 [main] INFO vCSc : Scanning Cluster ClientCluster for Host Systems 2023-06-16T18:17:29.583 [main] INFO vCSc : Host[0] : 192.168.1.253 : Connection connected : Power poweredOn : MaintenanceMode false : Active 2023-06-16T18:17:29.585 [main] INFO vCSc : 1 Active VMmark3 Hosts 2023-06-16T18:17:29.585 [main] INFO vCSc : 1 Hosts Under Test 2023-06-16T18:17:29.585 [main] INFO vCSc : Scanning for Provisioning Source VMs: 2023-06-16T18:17:29.595 [main] INFO vCSc : VM: ElasticDB0 Present 2023-06-16T18:17:29.604 [main] INFO vCSc : VM: ElasticAppB0 Present 2023-06-16T18:17:29.612 [main] INFO vCSc : VM: ElasticAppA0 Present 2023-06-16T18:17:29.621 [main] INFO vCSc : VM: Standby0 Present 2023-06-16T18:17:29.629 [main] INFO vCSc : VM: DS3DB0 Present 2023-06-16T18:17:29.636 [main] INFO vCSc : VM: AuctionWebB0 Present 2023-06-16T18:17:29.643 [main] INFO vCSc : VM: AuctionWebA0 Present 2023-06-16T18:17:29.652 [main] INFO vCSc : VM: ElasticWebB0 Present 2023-06-16T18:17:29.660 [main] INFO vCSc : VM: ElasticWebA0 Present 2023-06-16T18:17:29.668 [main] INFO vCSc : VM: AuctionMSQ0 Present 2023-06-16T18:17:29.676 [main] INFO vCSc : VM: Client0 Present 2023-06-16T18:17:29.687 [main] INFO vCSc : VM: AuctionLB0 Present 2023-06-16T18:17:29.696 [main] INFO vCSc : VM: AuctionNoSQL0 Present 2023-06-16T18:17:29.703 [main] INFO vCSc : VM: AuctionAppB0 Present 2023-06-16T18:17:29.710 [main] INFO vCSc : VM: DS3WebC0 Present 2023-06-16T18:17:29.718 [main] INFO vCSc : VM: AuctionAppA0 Present 2023-06-16T18:17:29.726 [main] INFO vCSc : VM: DS3WebB0 Present 2023-06-16T18:17:29.734 [main] INFO vCSc : VM: ElasticLB0 Present 2023-06-16T18:17:29.741 [main] INFO vCSc : VM: DS3WebA0 Present 2023-06-16T18:17:29.751 [main] INFO vCSc : VM: AuctionDB0 Present 2023-06-16T18:17:29.751 [main] INFO vCSc : Provision Source VMs Present : Environment Ready for Scale Out of VMs 2023-06-16T18:17:29.751 [main] INFO vCSc : Identifying Cluster Hosts' Networking: Provisioning NetworkLabel 'VM Network' Type standard 2023-06-16T18:17:29.770 [main] INFO vCSc : Reviewing host portgroups : 192.168.1.253 2023-06-16T18:17:29.770 [main] INFO vCSc : 192.168.1.253: 'VM Network' NetworkLabel : 'vSwitch0' vSwitch: Target NetworkLabel 2023-06-16T18:17:29.770 [main] INFO vCSc : 192.168.1.253: 'Management Network' NetworkLabel : 'vSwitch0' vSwitch 2023-06-16T18:17:29.770 [main] INFO vCSc : Provisioning NetworkLabel VM Network Found on All Hosts 2023-06-16T18:17:29.770 [main] WARN vCSc : Skipping Network tests as TemplateVM is null 2023-06-16T18:17:29.770 [main] INFO MAIN : Provisioning Validation Passed 2023-06-16T18:17:29.771 [main] INFO RMQ : Starting RabbitMQ DeleteQueue 2023-06-16T18:17:29.809 [main] INFO RMQ : Deleted Queue vmmark 2023-06-16T18:17:29.812 [main] INFO vCSc : Cataloging Deployed VMs (1 Tiles) 2023-06-16T18:17:29.830 [main] INFO vCSc : IP:Name :: 192.168.1.11 ElasticDB0 2023-06-16T18:17:29.845 [main] INFO vCSc : IP:Name :: 192.168.1.10 ElasticAppB0 2023-06-16T18:17:29.860 [main] INFO vCSc : IP:Name :: 192.168.1.9 ElasticAppA0 2023-06-16T18:17:29.874 [main] INFO vCSc : IP:Name :: 192.168.1.20 Standby0 2023-06-16T18:17:29.889 [main] INFO vCSc : IP:Name :: 192.168.1.5 DS3DB0 2023-06-16T18:17:29.905 [main] INFO vCSc : IP:Name :: 192.168.1.15 AuctionWebB0 2023-06-16T18:17:29.919 [main] INFO vCSc : IP:Name :: 192.168.1.14 AuctionWebA0 2023-06-16T18:17:29.933 [main] INFO vCSc : IP:Name :: 192.168.1.8 ElasticWebB0 2023-06-16T18:17:29.946 [main] INFO vCSc : IP:Name :: 192.168.1.7 ElasticWebA0 2023-06-16T18:17:29.960 [main] INFO vCSc : IP:Name :: 192.168.1.13 AuctionMSQ0 2023-06-16T18:17:29.974 [main] INFO vCSc : IP:Name :: 192.168.1.1 Client0 2023-06-16T18:17:29.988 [main] INFO vCSc : IP:Name :: 192.168.1.12 AuctionLB0 2023-06-16T18:17:30.003 [main] INFO vCSc : IP:Name :: 192.168.1.18 AuctionNoSQL0 2023-06-16T18:17:30.017 [main] INFO vCSc : IP:Name :: 192.168.1.17 AuctionAppB0 2023-06-16T18:17:30.030 [main] INFO vCSc : IP:Name :: 192.168.1.4 DS3WebC0 2023-06-16T18:17:30.043 [main] INFO vCSc : IP:Name :: 192.168.1.16 AuctionAppA0 2023-06-16T18:17:30.058 [main] INFO vCSc : IP:Name :: 192.168.1.3 DS3WebB0 2023-06-16T18:17:30.071 [main] INFO vCSc : IP:Name :: 192.168.1.6 ElasticLB0 2023-06-16T18:17:30.084 [main] INFO vCSc : IP:Name :: 192.168.1.2 DS3WebA0 2023-06-16T18:17:30.104 [main] INFO vCSc : IP:Name :: 192.168.1.19 AuctionDB0 2023-06-16T18:17:30.106 [main] INFO vCSc : All 20 VMs Cataloged Successfully 2023-06-16T18:17:30.107 [main] INFO MAIN : Scanning for Starting Tile: 20 VMsPerTile 2023-06-16T18:17:30.107 [main] ERROR MAIN : Starting Tile 1 : ProvisioningNumTiles 1 : No work to be done, delete VMs if you wish to recreate them or increase the ProvisionNumTiles value 2023-06-16T18:17:30.111 [main] ERROR MAIN : Exception Caught: Exiting Early: No work to be done, delete VMs if you wish to recreate them or increase the ProvisionNumTiles value : See Log for details : exitSetting true 2023-06-16T18:17:30.111 [main] INFO MAIN : Email Disabled 2023-06-16T18:17:30.111 [main] INFO MAIN : Ending Early  
Access to the VMmark template has been restored.  
Hi @john72 , this board is only to support VMmark, a VMware benchmark.
Hi I wonder this is right channel to ask this Do we have VMC Dimension performance of NS/EW traffic announsed to customers? THx  
@Virtualboy1 This particular board is for assistance with the VMmark benchmark only.  I would suggest posting this under ESXi Discussions - VMware Technology Network VMTN instead. Thanks, David
So I have been suspecting storage latency issues in our VMware environment for a while. So been working on ways to find solution to the problem. Core issue has been ESXi host disconnecting from vCent... See more...
So I have been suspecting storage latency issues in our VMware environment for a while. So been working on ways to find solution to the problem. Core issue has been ESXi host disconnecting from vCenter server and getting greyed out. When this happens we can't do much with the host or VM's on it except for shutting them down and starting them on different ESXi host. As a result I've been monitoring things using ESXTOP while I work on remediation and understand bottlenecks. Before DAVG figures were spiking above 25. So apparently anything above 25 is bad and indicates storage latency. However today when I was monitoring one of the host I came across these crazy figures for DAVG. And to me those looks bad. Now I understand I can't just look at DAVG and need to look at bigger picture. Can someone please look at attached and let me know how bad this is? Are spikes like this in DAVG normal? Or should it ideally not got this high even for a spike? More about environment below. VMware ESXi, 7.0.3, 20842708 vCenter 7.0.3 21477706 Dell PowerEdge hardware Dell Storage Center SCSI storage array iSCSI protocol being used to connect to storage array using software iSCSI adapter Backup runs mostly through the day so there is no definite time when backup runs or doesn't run Also have Zerto which is used for replication purpose Finally we also have live volumes at storage array layer We're looking at things like queue depth at iSCSI vmk port, making sure round robin is used for path selection etc. I must mention all firmware and drivers are updated recently already. This has been double checked already including compatibitlity
all the components are retails SKUs. Can i get exact recipe that you are using , i will try same from my end . HPE : SSP/SPP (Firmware ) ESXi OS : ESXi Image & source path Ajay, As @jamesz08 ... See more...
all the components are retails SKUs. Can i get exact recipe that you are using , i will try same from my end . HPE : SSP/SPP (Firmware ) ESXi OS : ESXi Image & source path Ajay, As @jamesz08 mentioned above, there isn't a recipe per se. HPE BIOS/System ROM: 1.20 ESXi OS: ESXi 7.0 U3 Apr 2023  Also, please try using an unmodified VMmark3 template (in other words, do not install a specific version of VMware Tools), just to remove that as a potential variable. Thanks, David
i meant to say test servers , server components are retails SKUs confirmed from my colleague
Hi Ajay, Earlier in the thread you mentioned "these are beta servers".  Can you clarify exactly which component(s) in the servers are beta? Thanks, David
all the components are retails SKUs. Can i get exact recipe that you are using , i will try same from my end .   HPE : SSP/SPP (Firmware ) ESXi OS : ESXi Image & source path   Thanks & Regards ... See more...
all the components are retails SKUs. Can i get exact recipe that you are using , i will try same from my end .   HPE : SSP/SPP (Firmware ) ESXi OS : ESXi Image & source path   Thanks & Regards Ajay vaaka
If these are beta CPUs I suggest contacting HPe to update the systems to production CPUs.  This kernel panic does not happen on any production Zen4 system I or our other partners have run on.
these are beta servers . WIndows vm's running fine on it
There is no real recipe.  I just installed the 7.0 HPe image on the system and booted the Vmmark3 template.   Are other VMs booting OK on the system?  These are production CPUs (not beta or preprodu... See more...
There is no real recipe.  I just installed the 7.0 HPe image on the system and booted the Vmmark3 template.   Are other VMs booting OK on the system?  These are production CPUs (not beta or preproduction)?
can you please share your receipe . i will reproduce on my end .   Here the BIOS settings i'm using    
Have you made any BIOS changes in the system? I cannot reproduce this on our system, so I am not sure what other help I can give you.
Thanks for the update
we use NFS storage, storage is not the issue here. I moved the same template to Intel hardware and hosted it on the same datastore. it is up and running. Moved to local VMFS, still the issue is pers... See more...
we use NFS storage, storage is not the issue here. I moved the same template to Intel hardware and hosted it on the same datastore. it is up and running. Moved to local VMFS, still the issue is persistent. I have two servers with AMD hardware, same issue on both. I don't see any DIMM issues as well.    Thanks & Regards Ajay Vaaka    
That is the default tools version which should work. Are you running from netapp storage?  Can you move the VM to another storage to see if it still occurs?  Otherwise I can only think the host has ... See more...
That is the default tools version which should work. Are you running from netapp storage?  Can you move the VM to another storage to see if it still occurs?  Otherwise I can only think the host has some hardware issues such as a bad DIMM.  
VMware tools : 9.10.2.48224 ( build-2822639) is running on it . will update it now.   i tried BIOS to v1.2 , kernel issue is persistent
I verified the VM boots OK with the April HPe custom image.  I do not have any Netapp storage, so I cannot test with that installed. What version of tools are you trying to use?  Did you update the ... See more...
I verified the VM boots OK with the April HPe custom image.  I do not have any Netapp storage, so I cannot test with that installed. What version of tools are you trying to use?  Did you update the tools inside the template?