Significant Imbalance between NUMA nodes Detected

[Gooose]

Hi Guys,

I'm currently running ESX 4.0 (Update2) in an HA cluster consisting of 4 servers attached to a Netapp Storage appliance utilising NFS mount points.

The ESX Servers are running on HP DL580 G7, and originally had 128GB RAM per server.

At the weekend, I installed an additional 64GB per server (8 x 8GB dual rank sticks) however I'm now getting the following message from the console:

cpu0:0)NUMA: 837: Significant imbalance between NUMA nodes detected. Performance may be impacted.

I believe the reason for this is the HP server has 4 available memory banks, of which I have two consisting of 32GB, and the other two consisting of 64GB giving me 192Gb in total.  The 32GB banks are made up of 4GB dual rank stick which came with the server.

Obviously, the imbalance is due to the fact that 2 CPU's are receiving less memory than the other two.

In an attempt to resolve the matter, I reallocated the RAM so that each bank had 48GB, however following this change the HP server failed to boot.

As far as I can make out, the message does not appear to be having any detrimental affect on the environment, and the easiest way to resolve the matter would be to replace the 4GB stick with 8GB stick to balance the memory banks?

Another change I have tried is to set the NUMA.REBALANCE advanced setting in VMware to disabled, again this did not clear the message.

Numa.RebalanceEnable and set to 0 to disable.

Has anyone ever come across this message before and managed to resolve it?  As mentioned the HP server itself is recognising the full allocation of 192GB RAM, and so is VMware.  From the VI client no error messages are showing, it's just purely from the console?

Any help would be greatly appreciated.

Thanks

[a_p_]

The "imbalanced" message is expected with different RAM population on the CPUs. Best practice is to populate the DIMMs evenly across all CPUs. Please take a look at the HP quickspecs (http://h18000.www1.hp.com/products/quickspecs/13669_na/13669_na.html) --> "DIMM installation guidelines"

André

[Gooose]

Morning Andre, thanks for the quick response, and the guide which is very useful.

So basically, the only way I can resolve the message is by ensuring that each memory banks contains the same allocation?

Will I see much of a performance hit if I leave the setup as it is?  Our environment is not stressed infact as a whole it's only running at 50% utilisation.

Many thanks for your help

[a_p_]

So basically, the only way I can resolve the message is by ensuring that each memory banks contains the same allocation?

Yes, at least as I know. I'm in the lucky position to usually being involved in my client's hardware planning and memory is one of the things I keep an eye on.

Will  I see much of a performance hit if I leave the setup as it is?  Our  environment is not stressed infact as a whole it's only running at 50%  utilisation.

Well, I can't tell you for sure. I don't think there will be a huge difference in performance, but I can't give you any facts. Never tested this.

André