Installing KNEM

KNEM configures, builds and installs as usual.

 $ ./configure --prefix=/opt/knem
 $ make
 $ make install

If building from a GIT glone, you may need to install automake >=1.10 and autoconf >=2.61, and then prepare the script before configure:

 $ ./autogen.sh

Everything is installed in the prefix directory (/opt/knem by default), including the kernel module and udev rules file. If you want modprobe and udev to properly handle KNEM, the relevant files should be installed in /lib/modules and /etc/udev/rules.d instead. To do so:

 $ /path/to/knem/install/sbin/knem_local_install

Usually, make install will run only once per cluster and install in a shared directory, while knem_local_install will run once per machine since /lib/modules and /etc/udev/rules.d are not shared between machines.

If you ever want to uninstall KNEM, you may run:

 $ rmmod knem
 $ /path/to/knem/install/sbin/knem_local_install --uninstall
 $ make uninstall

It is also possible to build KNEM as a RPM package which bundles the result of both make install and knem_local_install:

 $ rpmbuild -tb knem-<version>.tar.gz

KNEM also supports DKMS (Dynamic Kernel Module Support) which lets the system automatically rebuild and install modules for new kernels. See dkms.conf for details.

Starting KNEM

To start KNEM, assuming you ran knem_local_install, just run:

 $ modprobe knem

By default, udev will set the /dev/knem device group to rdma. Thus any KNEM user should be added to this group first.

If you did not run knem_local_install, you may still load the kernel module and setup the device group manually:

 $ insmod /path/to/knem/install/lib/modules/$(uname -r)/knem.ko
 $ chgrp rdma /dev/knem

If the rdma group does not exist on your machine, you may modify the udev rule to set the mode to 0666 and not to change the file group.

Configuring your MPI implementation to use KNEM

MPICH2 supports KNEM since version 1.1.1 (it is therefore also supported by MVAPICH2 1.5). To use it, you should add the following options to the configure line:

 --with-nemesis-local-lmt=dma --with-knem=/path/to/knem/install/

It will use KNEM starting at 64kB messages, and offload copies to DMA engine after 2MB. To change the latter threshold, use the environment variable MPICH_NEM_LMT_DMA_THRESHOLD.

Open MPI supports KNEM since version 1.5.0. To use it, you should add the following option to the configure line:

 --with-knem=/path/to/knem/install/

KNEM will be used at runtime after 4kB. You may change this threshold with --mca btl_sm_eager_limit 32768. You may also disable KNEM entirely at runtime with --mca btl_sm_use_knem 0. KNEM will not offload copies to DMA engine by default. You should use --mca btl_sm_knem_dma_min 1048576 to enable copy offload starting at 1MB.

To check that KNEM is properly used, see Getting Statistics. Make sure your users are members of the rdma group so that they are granted access to the /dev/knem device file.

If you want to port your MPI implementation to KNEM, see the interface documentation.

Getting Statistics

To check that KNEM is properly used, how it processed requests, why some of them failed, you may read statistics:

  $ cat /dev/knem
  [...]
   Requests submitted                           : 10243
   Requests processed (total)                   : 10236
            processed (using DMA)               : 0
            processed (offloaded to thread)     : 0
  [...]
   Requests rejected (invalid flags)            : 5
            rejected (not enough memory)        : 2
  [...]
   Requests failed during memcpy from/to user   : 1
            failed during DMA copy              : 0

These statistics may also be cleared by a privileged user:

  # echo 1 > /dev/knem

If the knem module was loaded with parameter statsverbose=1 (appended to the insmod or modprobe command-line), reading from /dev/knem will also list the currently open memory regions for the current process (and even from all processes if the user is privileged).

  # cat /dev/knem
  [...]
   Context 5d343d48
    Region 5d344ad1 SingleUse
    Listed 1 regions
   Context 5d343d49
    Listed 0 regions
   Listed 8 contexts