Turns out that with any more than lightly loaded modern KVM + libvirt host system that houses more than a single CPU, NUMA tuning is essential.
If you see symptoms like high system CPU time peaks on the host, and steal CPU time on the virtual instances (or ‘missing’ CPU time), chances are the system is spending too much time transferring data between NUMA nodes. Another tell tale sign is guests / VMs that get shot down by the host because of an OOM (Out of Memory) condition, while there seems to be a healthy margin of free/buffered/cached memory.
The problem here is that host system memory is distributed across available NUMA nodes. Certain cores are associated to certain nodes. With current x86_64 machines usually one physical processor package, typically containing multiple CPU cores, also holds one NUMA node. It helps performance a lot if a single virtual machine (or any old process) is contained within a single node. The below picture, output of hwloc-ls (apt-get install hwloc) gives an idea of how this is layed out.
For that reason you might want to pin guests to a defined subset of logical cores, that are all in the same node. The memory assigned to guests within a single NUMA node should also together, fit within the memory the node has available. Having a non NUMA aware guest, even if there is only one, with more memory assigned to it than one NUMA node has available, does not make a lot of sense. There is a big performance penalty because part of the assigned RAM when addressed, always needs to get transferred between nodes. If you need to do this anyway, have a look at e.g. http://docs.openstack.org/developer/nova/testing/libvirt-numa.html – you can make the guest itself NUMA aware so it can efficiently use memory from multiple NUMA cells.
Helpful tools here are numastat -m (apt-get install numactl), the hwloc suite (apt-get install numactl hwloc, see picture, and you will need X forwarding (ssh -X) if you want the graphics from a remote system with hwloc-ls), lscpu.
Peripherals such as disks and NICs are also NUMA node bound, which can be important for DMA performance.
Example numastat output:
root@host-system:~# numastat -m Per-node system memory usage (in MBs): Node 0 Node 1 Total --------------- --------------- --------------- MemTotal 64328.18 64510.16 128838.34 MemFree 10501.73 2769.72 13271.46 MemUsed 53826.45 61740.43 115566.88 Active 48528.07 55331.48 103859.55 Inactive 3082.79 4013.75 7096.54 Active(anon) 45496.47 51762.40 97258.87 Inactive(anon) 0.12 0.18 0.30 Active(file) 3031.60 3569.08 6600.68 Inactive(file) 3082.67 4013.56 7096.23 Unevictable 0.00 0.00 0.00 Mlocked 0.00 0.00 0.00 Dirty 0.50 0.30 0.80 Writeback 0.00 0.00 0.00 FilePages 6115.00 7584.15 13699.15 Mapped 13.54 6.84 20.38 AnonPages 45495.84 51761.65 97257.49 Shmem 0.73 0.94 1.67 KernelStack 6.70 4.98 11.69 PageTables 94.40 124.43 218.82 NFS_Unstable 0.00 0.00 0.00 Bounce 0.00 0.00 0.00 WritebackTmp 0.00 0.00 0.00 Slab 1462.05 1622.14 3084.19 SReclaimable 701.69 1071.01 1772.70 SUnreclaim 760.36 551.12 1311.49 AnonHugePages 1396.00 438.00 1834.00 HugePages_Total 0.00 0.00 0.00 HugePages_Free 0.00 0.00 0.00 HugePages_Surp 0.00 0.00 0.00
From virsh capabilities, example NUMA layout:
<topology> <cells num='2'> <cell id='0'> <memory unit='KiB'>65872056</memory> <cpus num='12'> <cpu id='0' socket_id='0' core_id='0' siblings='0,12'/> <cpu id='1' socket_id='0' core_id='1' siblings='1,13'/> <cpu id='2' socket_id='0' core_id='2' siblings='2,14'/> <cpu id='3' socket_id='0' core_id='3' siblings='3,15'/> <cpu id='4' socket_id='0' core_id='4' siblings='4,16'/> <cpu id='5' socket_id='0' core_id='5' siblings='5,17'/> <cpu id='12' socket_id='0' core_id='0' siblings='0,12'/> <cpu id='13' socket_id='0' core_id='1' siblings='1,13'/> <cpu id='14' socket_id='0' core_id='2' siblings='2,14'/> <cpu id='15' socket_id='0' core_id='3' siblings='3,15'/> <cpu id='16' socket_id='0' core_id='4' siblings='4,16'/> <cpu id='17' socket_id='0' core_id='5' siblings='5,17'/> </cpus> </cell> <cell id='1'> <memory unit='KiB'>66058400</memory> <cpus num='12'> <cpu id='6' socket_id='1' core_id='0' siblings='6,18'/> <cpu id='7' socket_id='1' core_id='1' siblings='7,19'/> <cpu id='8' socket_id='1' core_id='2' siblings='8,20'/> <cpu id='9' socket_id='1' core_id='3' siblings='9,21'/> <cpu id='10' socket_id='1' core_id='4' siblings='10,22'/> <cpu id='11' socket_id='1' core_id='5' siblings='11,23'/> <cpu id='18' socket_id='1' core_id='0' siblings='6,18'/> <cpu id='19' socket_id='1' core_id='1' siblings='7,19'/> <cpu id='20' socket_id='1' core_id='2' siblings='8,20'/> <cpu id='21' socket_id='1' core_id='3' siblings='9,21'/> <cpu id='22' socket_id='1' core_id='4' siblings='10,22'/> <cpu id='23' socket_id='1' core_id='5' siblings='11,23'/> </cpus> </cell> </cells> </topology>
Example (virsh edit instance) instance CPU/memory definition for the above layout:
<memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <vcpu placement='static' cpuset='8-9,20-21'>4</vcpu>
The logical CPUs in the cpuset are all within one NUMA node (cell with id 1) and the memory fits within the 64GB available to that node.