Friday, February 12, 2016

Few notes about network namespaces in Linux

For some time I'm working with network namespaces as implemented in the Linux kernel. Here I'll collect some notes about the implementation, behavior, usage and anything else I learn while using network namespaces.

Kernel API for NETNS

Kernel offers two system calls that allow management of network namespaces. The first one is for creating a new network namespace, unshare(2). Actually, this system call allows other types of namespaces to be created, but we are interested only in network namespaces. So, to create a new network namespace you should call the function like this:
#include <sched.h>
And that would define a new network namespace.

There are two ways other processes can now use that network namespace. The first approach is for the process that created new network namespace to fork other processes and each forked process woulds share and inherit, if exec is used, network namespace from the parent process.

The second approach is to use setns(2) system call. For that to work you have to have a file descriptor that somehow related to the network namespace you want to use. Again, there are two approaches how to obtain the file descriptor.

The first approach is to know the process that lives currently in the required network namespace. Let's say that the PID of the given process is $PID. So, to obtain file descriptor you should open the file /proc/$PID/ns/net file and that's it. This approach works always.

The second approach works only for iproute2 compatible tools. Namely, ip command when creating new network namespace creates a file in /var/run/netns directory and bind mounts new network namespace to this file. So, if you know a name of network namespace you want to access (let's say the name is NAME), to obtain file descriptor you just need to open(2) related file, i.e. /var/run/netns/NAME.

Note that there is no system call that would allow you to remove some existing network namespace. Each network namespace exists as long as there is at least one process that uses it, or there is a mount point.

Two remarks for the end of this section. First, there is no system call that would allow one process to move some other process into another network namespace! And second, you need appropriate privileges to use the mentioned system calls, i.e. regular user processes can't switch namespaces.

Socket API behavior

The next question is how Socket API behaves when network namespaces are used, and things here are quite interesting.

First, each socket handle you create is bound to whatever network namespace was active at the time the socket was created. That means that you can set one network namespace to be active (say NS1) create socket and then immediately set another network namespace to be active (NS2). The socket created is bound to NS1 no matter which network namespace is active and socket can be used normally. In other words, when doing some operation with the socket (let's say bind, connect, anything) you don't need to activate socket's own network namespace before that!

Also, to note is that network namespace is per-thread setting, meaning if you set certain network namespace in one thread, this won't have any impact on other threads in the process.

Command line tools

There are two command line tools available to manipulate network namespaces. The first one is nsenter(1) which isn't specific to networking. It allows one to start some process within predefined network namespace. The second tool is ip command from iproute2 package. It allows management of network namespaces and also allows network interfaces to be switched between different namespaces.

NETLINK behavior


Tuesday, February 2, 2016

NetworkManager architecture

After figuring out how VPN establishment works in NetworkManager the next big question is about the architecture of NetworkManager and its run-time behavior. This post tries to provide answer to the given question. This post consists of two parts. The first part, Source code organization, gives information about the source tree organization of NetworkManager. The second part, Run-time initialization behavior has a goal to describe initialization of NetworkManager and its key parts. Note that I'll update this post as I learn about NetworkManager.

The last update time of this post was: February 2nd, 2016.

Source code organization

So, let us start with a source tree organization. If you take a look at the top level directory of the NetworkManager git tree, you'll find there some files and the set of directories. The files are build scripts and configuration files, there are also some documentation files. But, in the end all the interesting functionality is placed within the top level directories. There are the following top level directories:
  • callouts

  • clients

    Clients to control NM like nmcli or nmtui.
  • introspection

    contains XML description of DBus interfaces provided by NetworkManager and its components. For example, VPN plugins are external to NetworkManager but they use DBus to communicate with NetworkManager and there is an XML file that describes VPN plugin's interfaces. All the descriptions are converted into C code stubs and skeletons using dbus-binding-tool. There is a lot of magic staff happening here.
  • libnm

    Libraries intended to be used by clients accessing and using services provided by NetworkManager. This also includes application that control NetworkManager itself (like nmcli and nmtui)
  • libnm-core

    Core NM libraries used by NetworkManager itself and all the other components, like command line utilities.
  • libnm-gliblibnm-util

    Old NM libraries that are deprecated starting with 1.0 release of NetworkManager. Still, it seems that at least some functionality is used from those libraries so probably until all the pieces of NetworkManager are converted to the new libraries they are to stay. Note that in my posts I ignore the content of those libraries as much as possible and I concentrate on the previous two libraries, libnm and libnm-core.
  • src

    The directory with the code specific to the NetworkManager unlike the previous directories that contain code used by different programs. The src/ directory contains file and  a number of subdirectories:
    • devices/

      Classes and sub classes used to represent different networking devices, real and virtual, that can be managed by NetworkManager.
    • dhcp-manager/

      Manager class that manages and interfaces DHCP clients to NetworkManager.
    • dns-manager/

    • dnsmasq-manager/

    • platform/

      Platform specific code that interfaces NetworkManager to particular operating system it is running on. For the moment there is a base class and derived Linux specific class.
    • ppp-manager/

    • rdisc/

      Code to monitor and parse IPv6 router advertisement messages.
    • settings/

      Code to keep track of connections. Since connections are read from configuration files that depend on particular distributions this directory also contains plugins, one plugin for each supported distribution.
    • supplicant-manager/

    • systemd/

    • vpn-manager/

      VPN specific code. Here is a manager class that is used to manage all the VPN connections and also class that is used to represent each active VPN connection.

Run-time initialization behavior

When NM is started execution starts from main() function that can be found in src/main.c. The goal of the main function is to parse command line options, check some prerequisites for running NM, do initialization steps and start glib main loop. The most interesting are initialization steps which are:
  • Initialize logging.
  • Create configuration object singleton, NM_TYPE_CONFIG (defined in file src/nm-config.c).
  • Creates a singleton for managing DBus connections and communication, NM_TYPE_BUS_MANAGER (defined in file src/nm-bus-manager.c).
  • Creates a singleton that is platform dependent used for management of network subsystem, NM_TYPE_LINUX_PLATFORM (defined in file src/platform/nm-linux-platform.c).
  • Creates a main singleton object of class NM_TYPE_MANAGER (defined in file src/nm-manager.c).
  • Initializes loopback interface.
So, while NetworkManager is running it basically consists of a set of singleton objects that communicate and make what is named NetworkManager.


Wednesday, January 13, 2016

Connections in NetworkManager

Connections, as defined and used by NM, are very close to PvDs. The goal of this post is to analyse data structures/functions for connections within NetworkManager and so that plan of integrating PvDs into NM can be developed. This is done in a separate post.

A definition of connection in NetworkManager can be found in the comment within the libnm-core/nm-connection.c file:
An #NMConnection describes all the settings and configuration values that are necessary to configure network devices for operation on a specific network. Connections are the fundamental operating object for NetworkManager; no device is connected without a #NMConnection, or disconnected without having been connected with a #NMConnection.

Each #NMConnection contains a list of #NMSetting objects usually referenced by name (using nm_connection_get_setting_by_name()) or by type (with nm_connection_get_setting()). The settings describe the actual parameters with which the network devices are configured, including device-specific parameters (MTU, SSID, APN, channel, rate, etc) and IP-level parameters (addresses, routes, addressing methods, etc).
In the following text we'll see how connections are implemented in the NM code, how they are initialized and how they are accessed over the DBus. Note that there are some parts related to connections that are specific for a system/distribution on which NM is running. In that case I concentrate on how things are done on Fedora (and very likely all RHEL derivatives).

Class and interface hierarchy

The base for all connection objects in NetworkManager is the interface defined in the files libnm-core/nm-connection.[ch]. The interface is implemented by the following classes:
  • Class NMSettingsConnectionClass defined in the files  src/settings/nm-settings-connection.[ch].

    This class is used by NetworkManager daemon and it is exported via DBus interface.
  • Class NMRemoteConnectionClass defined in the files libnm/nm-remote-connection.[ch].

    Used by clients in clients subdirectory.
  • Class NMSimpleConnectionClass defined in the files libnm-core/nm-simple-connection.[ch].

    This is the object passed via DBus so it is for communicating connections from and to NetworkManager and its clients.

Accessing individual connection data stored in NetworkManager

Each connection, active or not, known to the NetworkManager is exposed through DBus on path /org/freedesktop/NetworkManager/Settings/%u where %u is a sequence number assigned to each connection. Interface implemented by each connection is org.freedesktop.NetworkManager.Settings.Connection. The interface is described in introspection/nm-settings-connection.xml file.

To invoke a method defined in interface on the given object you can use dbus-send command line tool like this:
dbus-send --print-reply --system \
    --dest=org.freedesktop.NetworkManager \
    /org/freedesktop/NetworkManager/Settings/0 \
In this particular case we are invoking GetSettings method on object /org/freedesktop/NetworkManager/Settings/0 which will return us its configuration parameters. Note that invoking this particular method is easy since there are no arguments to the method.

The class that represents connection, and that is answering to DBus messages, is declared in src/settings/nm-settings-connection.[ch] files. This class implements interface NM_TYPE_CONNECTION and also subclasses NM_TYPE_EXPORTED_OBJECT class.  The NM_TYPE_EXPORTED_OBJECT class has all the methods necessary to expose the object on DBus.

To see what functions are called when DBus methods are called take a look at the end of the source file nm-settings-connection.c. There, you'll find the following code:
nm_exported_object_class_add_interface (NM_EXPORTED_OBJECT_CLASS (class),
    "Update", impl_settings_connection_update,
    "UpdateUnsaved", impl_settings_connection_update_unsaved,
    "Delete", impl_settings_connection_delete,
    "GetSettings", impl_settings_connection_get_settings,
    "GetSecrets", impl_settings_connection_get_secrets,
    "ClearSecrets", impl_settings_connection_clear_secrets,
    "Save", impl_settings_connection_save,
What this code does is that it binds GBus methods to function that should be called. When we called GetSettings method, obviously that ended up in the function impl_settings_get_settings().

The first step done when processing GetSettings method is authorization check. After authorization check has succeeded, the return message is constructed in get_settings_auth_cb() method.

Accessing and manipulating all connections in NetworkManager

NetworkManager exposes interface org.freedesktop.NetworkManager.Setting on object org.freedesktop.NetworkManager.Setting that, among other things, allows the user to retrieve list of all the known connections to the NetworkManager.  To get all connections you would could use the following dbus-send command:
dbus-send --print-reply --type=method_call --system \
        --dest=org.freedesktop.NetworkManager \
        /org/freedesktop/NetworkManager/Settings \
        org.freedesktop.DBus.Properties.Get \
        string:org.freedesktop.NetworkManager.Settings \
This would get you something like the following output:
variant array [
   object path "/org/freedesktop/NetworkManager/Settings/10"
   object path "/org/freedesktop/NetworkManager/Settings/11"
   object path "/org/freedesktop/NetworkManager/Settings/12"
   object path "/org/freedesktop/NetworkManager/Settings/13"
   object path "/org/freedesktop/NetworkManager/Settings/14"
   object path "/org/freedesktop/NetworkManager/Settings/15"
   object path "/org/freedesktop/NetworkManager/Settings/0"
   object path "/org/freedesktop/NetworkManager/Settings/1"
   object path "/org/freedesktop/NetworkManager/Settings/2"
   object path "/org/freedesktop/NetworkManager/Settings/3"
   object path "/org/freedesktop/NetworkManager/Settings/4"
   object path "/org/freedesktop/NetworkManager/Settings/5"
   object path "/org/freedesktop/NetworkManager/Settings/6"
   object path "/org/freedesktop/NetworkManager/Settings/7"
   object path "/org/freedesktop/NetworkManager/Settings/8"
   object path "/org/freedesktop/NetworkManager/Settings/9"
   object path "/org/freedesktop/NetworkManager/Settings/16"
The exact output will depend on your particular setup and usage.

The given interface and property is implemented by class NMSettingsClass (defined in src/settings/nm-settings.[ch]). This class implements interface NM_TYPE_CONNECTION_PROVIDED (defined in src/nm-connection-provider.[ch]). There is only one object of this class in NetworkManager and it is instantiated when NetworkManager is starting.

Looking in the file src/settings/nm-settings.c you can see at the end registration of function to be called when DBus messages are received. DBus interface of this module is defined in introspection/nm-settings.xml file. Here is the relevant code that binds DBus methos to the functions that implement them:
nm_exported_object_class_add_interface (
        "ListConnections", impl_settings_list_connections,
        "GetConnectionByUuid", impl_settings_get_connection_by_uuid,
        "AddConnection", impl_settings_add_connection,
        "AddConnectionUnsaved", impl_settings_add_connection_unsaved,
        "LoadConnections", impl_settings_load_connections,
        "ReloadConnections", impl_settings_reload_connections,
        "SaveHostname", impl_settings_save_hostname,
So, when we called ListConnections method, obviously that ended up in the function impl_settings_list_connections(). Here, we'll emphasize one more method, LoadConnections. This DBus method, implemented in impl_settings_load_connections(), load all connections defined in the system. We'll take a look now at that method.

Initializing connections

All network connections are loaded and initialized from two sources: system dependent network configuration and VPN configuration scripts.

System dependent network configuration

There are several types of distributions with different network configuration mechanisms. Since that part is obviously system dependent, NetworkManager has a plugin system that isolates the majority of NetworkManager code from those system dependent parts. Plugins can be found in the directory src/settings/plugins. Additionally, all the plugins are based on the src/settings/plugin.[ch] base class. In the case of Fedora Linux (as well as RHEL, CentOS and other derivatives) network configuration is recorded in scripts in the directory /etc/sysconfig/network-scripts. and the plugin that handles those configuration files is stored in the directory src/settings/plugins/ifcfg-rh.

The initialization of connections stored in the directory /etc/sysconfig/network-scripts is done when NetworkManager bootstraps and instantiates object NMSettings of type NMSettingsClass. This is performed in the function nm_manager_start() in the file src/nm-manager.c. There, the method nm_settings_start() is called on the NMSettings object which in turn first initializes all plugins (as, by default, found in the directory /usr/lib64/NetworkManager/). It then calls private method load_connections() to actually load all connections. Note that in the directory  /usr/lib64/NetworkManager/ there are plugins of other types too, but only plugins that have prefix libnm-settings-plugin- are loaded. Which plugins should be loaded are can be defined in three places (lowest to highest priority):

  1. Compile time defaults, as given to script, or, by default for RHEL type systems "ifcfg-rh,ibft" plugins.
  2. In configuration file /etc/NetworkManager/NetworkManager.conf.
  3. As specified in the command line.

Method load_connections() iterates over every defined plugin and asks each plugin for all registered connections it knows by calling a method get_connections() within a specific plugin. For RedHat type of distributions the the plugin that handles all connections is src/nm-settings/plugins/ifcfg-rh/plugin.c and in that file function get_connections() is called. Now, if called for the first time, this function will in turn call read_connections() within the same plugin/file that will read all available connections. Basically, it opens directory /etc/sysconfig/network-scripts and builds a list of all files in the directory. Than, it tries to open each file and only those that were parsed properly are left as valid connections. Each found connection is stored in object NMIfcfgConnection of type NMIfcfgConnectionClass. These objects are defined in files src/nm-settings/plugins/ifcfg-rh/nm-ifcfg-connection.[ch].

When all the connections were loaded, read_connections() returns a list of all known connections to the plugin. The function load_connections() then, for each connection reported by each plugin, calls claim_connection() method in nm-connection.c. This function, among other tasks, exports the connection via DBus in a form described above, in the section Accessing individual connection data stored in NetworkManager.

VPN configuration scripts

Details about VPN connections are stored in /etc/NetworkManager/system-connections directory, one subdirectory per VPN. Those files are read by src/vpn-manager/nm-vpn-manager.c when the object is initialized and as such initialized when VPN manager is initialized. VPN manager also also monitors changes in the VPN configuration directory and acts appropriately.

Properties of a connection

Each connection has a set of properties attached to it in a form of a key-value pairs.

Activating a connection

A connection is activated by calling ActivateConnection DBus method. This method is implemented in the NetworkManager's main class/object, NMManager. This class/object is a singleton object who's impementation is in src/nm-manager.[ch] files. Looking at the code that binds DBus methods to the functions that implement them we can see that ActivateConnection is implemented by the function impl_manager_activate_connection(). The ActivateConnection method, and its implementation function, accept several parameters:
  1. Connection that should be activated identified by its connection path (e.g. "/org/freedesktop/NetworkManager/Settings/2").
  2. Device on which connection should be activated identified by its path (e.g. "/org/freedesktop/NetworkManager/Devices/2").
  3. Specific object?
Some of the input argument can be unspecified. To make them unspecified in DBus call they should be set to "/" and this will be translated into NULL pointer in the impl_manager_activate_connection() function. Of all combinations of parameters (with respect to being NULL or non-NULL) the following ones are allowed:
  1. When connection path is not specified device must be given. In that case all the connections for that device will be retrieved and the best one will be selected. "The best one" is defined as the most recently used one.
  2. If connection path is specified, then device might or might not be specified. In case it is not defined the best device for the given connection will be selected. To determine "the best device" first list of all devices is retrieved and then for each device status is checked (must be managed, available, compatible with the requested connection). Note that "compatible with the requested connection" means, for example, you can not start wireless connection on a wired connection.
There are "software only", or virtual, connections. Those are checked in the function nm_connection_is_virtual() which is implemented in the file libnm-core/nm-connection.c. When this post was written, the following connections were defined as virtual, or software only:
  • Bond
  • Team
  • Bridge
  • VLAN
  • TUN
  • IPtunnel
Finally, there are also VPN connections that also don't have associated devices.

When all the checks are performed, devices and connections are found, then an object of type NMActiveConnection is created in the function _new_active_connection(). Here, in case VPN connection is started, VPN establishment is initiated and you can read more about that process in another post.

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)