Philjans1
Contributor
Contributor

Trying to find a power issue for one of my VM

Hi,
I have a VM running a SQL database and I am trying to make sure the slowness users complains about is not coming from the VMware/Windows .
It is a 56 cores (before HT) 2x Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
Here's extop :

Philjans1_1-1698346302285.png

Philjans1_0-1698348266920.png

 

Philjans1_2-1698346347916.pngPhiljans1_3-1698346398023.png

And here's from the vmware.log:

2023-10-11T23:59:00.744Z In(05) vmx - hostCPUID name: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz

2023-10-11T23:59:00.787Z In(05) vmx - VMMonParseSchedConfig: Initial Vmx Memory Configuration
2023-10-11T23:59:00.787Z In(05) vmx - MemSched (MB): min: 12 sizeLimit: 1220 swapInitialFileSize: 89 prealloc: FALSE.
2023-10-11T23:59:00.787Z In(05) vmx - MemSched (KB): min: 11528 sizeLimit: 1249176 swapInitialFileSize: 91136 prealloc: FALSE.
2023-10-11T23:59:00.787Z In(05) vmx - MemSched (MB): paged: 449 PT total: 461 PT min: 5
2023-10-11T23:59:00.787Z In(05) vmx - MemSched (KB): paged: 459764 PT total: 471292 PT min: 4976
2023-10-11T23:59:00.787Z In(05) vmx - MemSched: properties preallocGuestMem: 0 pinned: 0 fpt: 0 ftCpt: 0 prealloc: 0 preallocPMem: 0
2023-10-11T23:59:00.787Z In(05) vmx - ls: [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,]
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: NUMA config: consolidation= 1 preferHT= 0 partitionByMemory = 0
2023-10-11T23:59:00.787Z In(05) vmx - numa: Hot add is enabled and vNUMA hot add is disabled, forcing UMA.
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: 10 VCPUs 1 VPDs 1 PPDs
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 0 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 1 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 2 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 3 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 4 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 5 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 6 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 7 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 8 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: VCPU 9 VPD 0 PPD 0 NodeMask ffffffffffffffff
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: 1 mem slices
2023-10-11T23:59:00.787Z In(05) vmx - numaHost: memSlice 0 PPD 0 - 0 BPN [ 0x10000000000 - 0x10000800000 )
2023-10-11T23:59:00.787Z In(05) vmx - llc: maximum vcpus per LLC: 28
2023-10-11T23:59:00.787Z In(05) vmx - llc: vLLC size: 2
2023-10-11T23:59:00.787Z In(05) vmx - CreateVM: Swap: generating normal swap file name.
2023-10-11T23:59:00.788Z In(05) vmx - FTCpt: (0 unk) vmk overheads: 0 pages (0 MB)
2023-10-11T23:59:00.788Z In(05) vmx - Swap file path: '/vmfs/volumes/63bed46b-fef0c8b9-3d87-00620b928ec0/SRVUNICOM/SRVUNICOM-8bd989e7.vswp'
2023-10-11T23:59:00.788Z In(05) vmx - Disk_GetDigestDiskCount: numDigestDisks = 0
2023-10-11T23:59:00.788Z In(05) vmx - Using numSMT = 1

How can we know what's normale/abnormal in the stats of extop?

And how does it looks?

tx!

Reply
0 Kudos
nordhuus
Contributor
Contributor

You are running 8 VM on 54vCPU? Have you tried to adjust the number of vCPU on the sql server? Have you also checked the performance on the storage?

Reply
0 Kudos
Philjans1
Contributor
Contributor

Before throwing cpu at issues, I like to investigate any problems first.
This is why I was hoping anyone could see issues with the extop stats I posted.

The SSD storage seems quite fast and no delays are seen in Windows PerfMon's

Reply
0 Kudos
lukaszzasko
Enthusiast
Enthusiast

Hi,

CPU Ready (%RDY) has big value which impact on vms. You can try to decrease number of vcpus on vms.

Philjans1
Contributor
Contributor

Hi,
Is there a well made simply explained of all the data it is providing?
Like the %RDY: what does it means and what are the best values for it?

Reply
0 Kudos
lukaszzasko
Enthusiast
Enthusiast

Hi,

CPU Ready Time – The time your VM is waiting in line to use the CPU on the host. CPU Ready can also be called the CPU Scheduler time, high time is bad – low time is good.

lukaszzasko_0-1698867243734.png

 

Reply
0 Kudos
Philjans1
Contributor
Contributor

Hi Lukaszzasko
Wmware said that %RDY and %CSTP where too high and asked to lower the vcpu counts

I have a server with 6 VMs where 3 of them are at 2vcores for 1 vsockets.

How to know how many vsockets and vcores to give them?
Right now:
TS03 has 12 vcores on 1 vsockets
TS02 has 20 vcores on 1 vsockets
Uni has 10 vcores on 1 vsockets.

This physical server has 2 cpu and a total of 56 cores.
There is 2 pnuma as per extop.

Also, some documentation says to always give 1 cores and just rise the number of vsockets to the desire numberr of cpu so that vnuma is activated and configures itself.
But other documentation says to always use just 1 vsockets and put as many vcores to it as required, and that way you avoid the numa barrier.

Other docs says to match the physical numa configs ... but do not explain what that means!

I'm tempted to lower those 3 vm to 8 vcores on 1 sockets eath and see the extop stats but "guessing" in a prod environment is not ideal especially if I turn off CPU Hot Plug so that vnuma doesn't get disabled... which means I have to wait a whole day and reboot the server the following night to re-ajust the level of vcpus

Any ideas?

Reply
0 Kudos
nordhuus
Contributor
Contributor

I agree with you to lower the number of vCPU for the test, but i would adjust to use two sockets since the physical server has 2 CPU's.

Reply
0 Kudos