This article assumes that there is a basic understanding of what NUMA locality is and how that relates to an ESX(i) host.
The tests below were performed with an ESXi host running version 4.1 Update 1, using 96 GB of memory and two pCPU’s. As shown below, there are two NUMA nodes. The virtual machines are running Windows Server 2003 x64 with 60 GB of vRAM.
Test #1 – One VM, 60 GB allocated & unused memory:
“NLMEM” in the screenshot above is the amount of NUMA local memory for the virtual machine. Even though the VM is configured for 60 GB of RAM within a NUMA node of only 48 GB, all of the memory is still classified as being “local” to the NUMA node that is VM is on. This is because the memory for the VM has only been allocated and not used – if all 60 GB were actually in use, then all of the memory would not be local to that VM’s home node because its home node can only support 48 GB.
Test #2 – Two 60GB VMs, allocated & unused memory:
The same results as test #1 – two VM’s, each residing on a separate NUMA home node and both are using 100% local memory even though their home node only supports 48 GB.
Test #3 – Two 60GB VMs, allocated and used memory:
To simulate memory being allocated and also used, I ran TestLimit using the “-d” parameter:
The results:
The “test_2003” virtual machine shows approximately 14 GB of remote memory, and 48 GB of local memory. Notice that home node 0 for the “test2003-2” VM still shows the entire 60 GB of memory as local to that VM, because it is not actually using the entire 60 GB of memory assigned to it. The NUMA load balancer is very sophisticated, and is able to account for allocated memory vs. memory in use to maintain an accurate account of the actual load of each VM.
Summary
Even though you should strive to maintain NUMA locality by appropriately sizing your VMs based off of the underlying infrastructure, using utilities like RESXTOP will not necessarily give you a complete picture of how the ESXi host is handling VM load. Take note of not only the memory allocated to the VMs, but also what the VMs are actually doing with those resources.
Seamless Computing 