Category Archives: Linux

CentOS, RedHat, Fedora: check_yum? check_updates!

Leave a reply

Since it took me days to discover the existence of an in-repo, alternative, let me post it here for people trying to do the same.

I’m used to having Icinga and NRPE monitor Ubuntu and Debian update status with check_apt (in repo: nagios-plugins-basic or monitoring-plugins-basic, depending on OS version). Was trying to do the same for yum based CentOS / RHEL but could only find out of repo check_yum.pl and check_yum.py (by Hari Sekhon, at https://github.com/HariSekhon/nagios-plugins/). Those work fine. However it turns out similar functionality is already available in-repository on at least CentOS 6.8 thru 7.2 and, googling around, likely also on Fedora and RHEL . There is a check_updates in package nagios-plugins-check-updates, which also works fine but might be hard to discover because of the name.

OpenVPN client connection not started on Ubuntu 16.04

Leave a reply

Ubuntu 16.04 has systemd as its init system. Usually getting an OpenVPN client configuration going, is a matter of dropping the .conf or .ovpn file together with key and certs, into /etc/openvpn. On Ubuntu 16.04, you can ‘service openvpn restart’ all you like, but no connection is being initiated, and the logs stay silent.

Solution:

  1. edit /etc/default/openvpn, uncomment AUTOSTART=”all”
  2. sudo systemctl daemon-reload
  3. sudo service openvpn restart

The comments still say “all” is the default, but that is no longer true in Ubuntu 16.04.

It is also possible to systemd-manage individual server/client configurations, in the style of ‘service openvpn@<my-config> start/stop/status’. See https://fedoraproject.org/wiki/Openvpn#Setting_up_a_Linux_OpenVPN_client.

 

GRUB not updating, showing old kernels in boot menu on Ubuntu 14.04

Leave a reply

Grub boot problem on Ubuntu 14.04 that people seem to have had for older versions too: the grub boot loader does not get updated with newly installed kernels. Did not find a solution on the net though, so here goes.

Newer kernels get installed but do not show up in the boot menu. A bit puzzling, non of update-grub, grub-install /dev/sda or adding and removing kernel packages made any difference. At boot there was still the same stubborn list of old kernels, and the new ones were not listed.

Turns out somehow grub (v1), Ubuntu package name ‘grub’ had ended up on my system, where the config was grub2 based. So I did apt-get remove grub; apt-get install grub-pc and all was well again, including the latest and greatest kernel in the boot menu.

Linux virtualization with libvirt: NUMA management

Leave a reply

Turns out that with any more than lightly loaded modern KVM + libvirt host system that houses more than a single CPU, NUMA tuning is essential.

If you see symptoms like high system CPU time peaks on the host, and steal CPU time on the virtual instances (or ‘missing’ CPU time), chances are the system is spending too much time transferring data between NUMA nodes. Another tell tale sign is guests / VMs that get shot down by the host because of an OOM (Out of Memory) condition, while there seems to be a healthy margin of free/buffered/cached memory.

The problem here is that host system memory is distributed across available NUMA nodes. Certain cores are associated to certain nodes. With current x86_64 machines usually one physical processor package, typically containing multiple CPU cores, also holds one NUMA node. It helps performance a lot if a single virtual machine (or any old process) is contained within a single node. The below picture, output of hwloc-ls (apt-get install hwloc) gives an idea of how this is layed out.

hwloc-ls-output

For that reason you might want to pin guests to a defined subset of logical cores, that are all in the same node. The memory assigned to guests within a single NUMA node should also together, fit within the memory the node has available. Having a non NUMA aware guest, even if there is only one, with more memory assigned to it than one NUMA node has available, does not make a lot of sense. There is a big performance penalty because part of the assigned RAM when addressed, always needs to get transferred between nodes. If you need to do this anyway, have a look at e.g. http://docs.openstack.org/developer/nova/testing/libvirt-numa.html – you can make the guest itself NUMA aware so it can efficiently use memory from multiple NUMA cells.

Helpful tools here are numastat -m (apt-get install numactl), the hwloc suite (apt-get install numactl hwloc, see picture, and you will need X forwarding (ssh -X) if you want the graphics from a remote system with hwloc-ls), lscpu.

Peripherals such as disks and NICs are also NUMA node bound, which can be important for DMA performance.

Example numastat output:

root@host-system:~# numastat -m

Per-node system memory usage (in MBs):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal                64328.18        64510.16       128838.34
MemFree                 10501.73         2769.72        13271.46
MemUsed                 53826.45        61740.43       115566.88
Active                  48528.07        55331.48       103859.55
Inactive                 3082.79         4013.75         7096.54
Active(anon)            45496.47        51762.40        97258.87
Inactive(anon)              0.12            0.18            0.30
Active(file)             3031.60         3569.08         6600.68
Inactive(file)           3082.67         4013.56         7096.23
Unevictable                 0.00            0.00            0.00
Mlocked                     0.00            0.00            0.00
Dirty                       0.50            0.30            0.80
Writeback                   0.00            0.00            0.00
FilePages                6115.00         7584.15        13699.15
Mapped                     13.54            6.84           20.38
AnonPages               45495.84        51761.65        97257.49
Shmem                       0.73            0.94            1.67
KernelStack                 6.70            4.98           11.69
PageTables                 94.40          124.43          218.82
NFS_Unstable                0.00            0.00            0.00
Bounce                      0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00
Slab                     1462.05         1622.14         3084.19
SReclaimable              701.69         1071.01         1772.70
SUnreclaim                760.36          551.12         1311.49
AnonHugePages            1396.00          438.00         1834.00
HugePages_Total             0.00            0.00            0.00
HugePages_Free              0.00            0.00            0.00
HugePages_Surp              0.00            0.00            0.00

From virsh capabilities, example NUMA layout:

    <topology>
      <cells num='2'>
        <cell id='0'>
          <memory unit='KiB'>65872056</memory>
          <cpus num='12'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,12'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,13'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,14'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,15'/>
            <cpu id='4' socket_id='0' core_id='4' siblings='4,16'/>
            <cpu id='5' socket_id='0' core_id='5' siblings='5,17'/>
            <cpu id='12' socket_id='0' core_id='0' siblings='0,12'/>
            <cpu id='13' socket_id='0' core_id='1' siblings='1,13'/>
            <cpu id='14' socket_id='0' core_id='2' siblings='2,14'/>
            <cpu id='15' socket_id='0' core_id='3' siblings='3,15'/>
            <cpu id='16' socket_id='0' core_id='4' siblings='4,16'/>
            <cpu id='17' socket_id='0' core_id='5' siblings='5,17'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>66058400</memory>
          <cpus num='12'>
            <cpu id='6' socket_id='1' core_id='0' siblings='6,18'/>
            <cpu id='7' socket_id='1' core_id='1' siblings='7,19'/>
            <cpu id='8' socket_id='1' core_id='2' siblings='8,20'/>
            <cpu id='9' socket_id='1' core_id='3' siblings='9,21'/>
            <cpu id='10' socket_id='1' core_id='4' siblings='10,22'/>
            <cpu id='11' socket_id='1' core_id='5' siblings='11,23'/>
            <cpu id='18' socket_id='1' core_id='0' siblings='6,18'/>
            <cpu id='19' socket_id='1' core_id='1' siblings='7,19'/>
            <cpu id='20' socket_id='1' core_id='2' siblings='8,20'/>
            <cpu id='21' socket_id='1' core_id='3' siblings='9,21'/>
            <cpu id='22' socket_id='1' core_id='4' siblings='10,22'/>
            <cpu id='23' socket_id='1' core_id='5' siblings='11,23'/>
          </cpus>
        </cell>
      </cells>
    </topology>

Example (virsh edit instance) instance CPU/memory definition for the above layout:

  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static' cpuset='8-9,20-21'>4</vcpu>

The logical CPUs in the cpuset are all within one NUMA node (cell with id 1) and the memory fits within the 64GB available to that node.

Ubuntu Server 14.04 installation: CD could not be mounted

Leave a reply

This happens a lot when trying to install Ubuntu Server 14.04 from USB stick (14.04.3 still has the issue).

[!! Detect and mount CD-ROM]
Your installation CD-ROM couldn't be mounted. This probably means that the CD-ROM was not in the drive. If so you can insert it and try again.
Retry mounting the CD-ROM?

.. and of course there is no CD-ROM at all, the installation media are on the USB drive. An easy way to work around this:

  1. Open a second console with Alt-F2 (get back to the installation dialogue with Alt-F1 later)
  2. Press enter to activate it
  3. Enter the following command to make /cdrom a softlink to /media:

    rmdir /cdrom; ln -s media /cdrom

  4. Go back to the installation process with Alt-F1
  5. Answer ‘yes’ to the question Retry mounting the CD-ROM?

From here on the installation should go on as normal.