Utilizing Keepalived Unicast for Kea 1.3.0 Site Failover

Utilizing Keepalived Unicast for Kea 1.3.0 Site Failover

ISC Kea 1.3.0 does not yet have built-in support for same or site failover, hot standby or an active/active load balanced setup. These HA features are on the roadmap for Kea 1.4.0.

This article will be discussing a multi site failover setup, which can relatively easily be solved by using Keepalived's unicast features combined with two bash scripts and a suitable network setup. Keepalived unicast VRRP is the best solution for intra subnet VRRP communication, since multicast does not traverse subnets without multicast routing.

In general this article could be seen as a PoC for creating a site failover solution for an udp service sharing a database.

The following illustration outlines the basic setup:

Kea 1.3.0 Site Failover

Network Requirements

Because the Kea servers are on two different sites, with different subnets, and no floating VIP via anycast is used, two DHCP helper adresses need to be configured on the network equipment. In this setup only one of those helper adresses will be responding at any given time.

Configuration Management

Since there is no virtual ip in this setup the interfaces-config differ between the two Kea servers. This part of the configuration could be handled by a configuration management system like Puppet. If possible I would recommend putting the subnet definitions and options under git revision and including it and hereby ensuring easier rollback and Kea syntax checking via git hooks. I will cover why combining Kea configuration managment and git is a sane solution in more detail in an upcoming article. Look here on how to achieve this with ISC DHCP.

Kea configuration and the database backend

The Kea servers are required to use a shared lease backend, such as MySQL.

The database is only accessed by one Kea server at a time, because of the active/passive nature of the setup. Setting up a fault tolerant database backend is a whole separate topic in itself, i.e a Percona Galera setup guide can be found here.

If you choose a MySQL lease backend I highly recommend using ProxySQL for its connection pooling and load balancing features.

A minimal configuration could look like this (here with DHCPv6):

{
    "Dhcp6": {
        "interfaces-config": {
            "interfaces": [
                "<network interface/ipv6 address specific for server>"
            ]
        },
        "control-socket": {
            "socket-type": "unix",
            "socket-name": "/tmp/kea-dhcp6-ctrl.sock"
        },
        "lease-database": {
            "type": "mysql",
            "name": "<database name>",
            "user": "<database user>",
            "password": "<mysql password>",
            "host": "<database server>",
            "port": 3306
        },
        "expired-leases-processing": {
            "reclaim-timer-wait-time": 10,
            "flush-reclaimed-timer-wait-time": 25,
            "hold-reclaimed-time": 3600,
            "max-reclaim-leases": 100,
            "max-reclaim-time": 250,
            "unwarned-reclaim-cycles": 5
        },
        "renew-timer": 1000,
        "rebind-timer": 2000,
        "preferred-lifetime": 3000,
        "valid-lifetime": 4000,
        "shared-networks": [
            {
                "name": "Test",
                "subnet6": [
                    {
                        "interface": "<network interface>",
                        "subnet": "<ipv6 subnet>",
                        "reservations": [
                            {
                                "hw-address": "<hw address>",
                                "ip-addresses": [
                                    "<reserved ip>"
                                ]
                            }
                        ],
                        "pools": [
                            {
                                "pool": "<pool range>"
                            }
                        ]
                    }
                ]
            }
        ]
    },
    "Logging": {
        "loggers": [
            {
                "name": "kea-dhcp6",
                "output_options": [
                    {
                        "output": "/var/log/kea-dhcp6.log"
                    }
                ],
                "severity": "INFO",
                "debuglevel": 0
            }
        ]
    }
}

As mentioned above, the configuration should be identical on both nodes, with the exception of the interface-config section.

Keepalived setup

The flow of events in case of failure looks like this:

Keepalived monitors the Kea process health on the local node and if the Kea health check fails; Keepalived immediately goes in to FAULT state.

The kea_check script also asks the witness server on a third site what the state of port 547 is on both Kea servers. The return values (1 for open 0 for closed) determines which action to take from this point on.

If the port state returned from the witness server is either master_alive or slave_alive the kea process check will run, else the script will exit with 1.

  1. If the master nodes Kea process is stopped or one of the tracked interfaces is down the, fallback node is promoted from BACKUP to MASTER state which in turn starts Kea.
  2. When the Keepalived Kea health check on the primary node succeeds it immediately becomes MASTER again. This is determined by the weight of the priority option.
  3. The fallback node is returned to BACKUP state again and Kea is killed on the fallback node.
  4. If the witness server returns that ports are up on both Kea servers indicating a intra site network partition, both Keepalived instances goes into FAULT state.
  5. If the witness server returns that ports are down on both Kea servers, both Keepalived instances goes directly into FAULT state.

These rules are not set in stone, as there are certainly scenarios where it would be beneficial to always start the service on the master when the witness reports ports are down on both nodes.

The above process should be easy to follow in the script below:

#!/bin/bash

# Remote kea_check possible states
master_alive=10
slave_alive=01
both_alive=11
both_dead=00
user_agent_secret='secretstuffmayn'

if ! check_return=$(curl --user-agent "$user_agent_secret" -s https://kea-witness-1:4777/check); then
   echo "curl failed"
   exit 1
fi

function kea_check {

  /bin/systemctl -q is-active kea-dhcp6

}

case $check_return in
  $master_alive)
    if grep -q BACKUP /var/run/keepalived.state; then
      if ! kea_check; then
        exit 0
      fi
    fi

    if grep -q MASTER /var/run/keepalived.state; then
      if ! kea_check; then
        exit 1
      fi
    fi
    ;;
  $slave_alive)
    if grep -q BACKUP /var/run/keepalived.state; then
      if ! kea_check; then
        exit 0
      fi
    fi

    if grep -q MASTER /var/run/keepalived.state; then
      if ! kea_check; then
        exit 1
      fi
    fi
    ;;
  $both_alive)
    exit 1
    ;;
  $both_dead)
    exit 1
    ;;
  *)
    echo "unknown return state"
    exit 1
esac

The Keepalived configuration itself:

global_defs {
   router_id aek
   smtp_server "<smtp server ip>"
   smtp_connect_timeout 30
   notification_email_from noreply@test.com
   notification_email {
        team@test.com
   }
}

vrrp_script kea_check  {
        script       "/usr/local/bin/kea_check.sh"
        interval 3
        fall 2
        rise 2
}

vrrp_instance unicast_instance {
        state MASTER
        interface "<interface name>"
        virtual_router_id "<number between 0-255>"
        priority 151
        advert_int 1
        track_interface {
               "<interface name 1>"
               "<interface name 2>"
               "<dummy interface>"
        }
        notify /usr/local/bin/keepalived_notify.sh
        track_script {
                kea_check
        }


        authentication {
                auth_type PASS
                auth_pass keaaek 
        }

        unicast_src_ip "<host ip>"

        unicast_peer {
            "<the peer ip>" 
        }
}

An important part of the configuration above is to set as many interfaces in the track_interface section as you have on the server, since this ensures that Keepalived goes in to FAULT state if any of them goes down. Some prefer to track a dummy interface as well in order to instigate a failover without stopping Keepalived.

The kea_check script interval is set to 3 seconds, since the udp check itself on the witness server take approximately 1 second.

The keepalived_notify.sh script keeps track of Keepalived state and starts or kills Kea depending on the state:

#!/bin/sh

TYPE=$1
NAME=$2
STATE=$3

service='kea-dhcp-6'

case $STATE in
        "MASTER") /bin/systemctl start $service
                  echo $STATE > /var/run/keepalived.state
                  exit 0
                  ;;
        "BACKUP") /usr/bin/pkill $service
                  echo $STATE > /var/run/keepalived.state
                  exit 0
                  ;;
        "FAULT") /usr/bin/pkill $service
                  echo $STATE > /var/run/keepalived.state
                  exit 0
                  ;;
        *)        echo "unknown state"
                  exit 1
                  ;;
esac

The Witness server

One more element is needed in order to avoid split-brain scenarios where a network partition between sites hinders VRRP communication between Keepalived peers. There are several possible solutions, where the most predominant ones can be found in the Heartbeat/Pacemaker stack. I choose a more simple route by using the Mojolicious (Perl) framework to program a simple port checker you can call via http(s).

This "Witness" service and port checker utility MUST be on a third site that is not a part of your own datacenters/network and MUST also be set up with a VIP and Keepalived.

A port check is clearly not enough to check the health of Kea, so MoWitness needs to be extended with a pseudo client which can renew a static lease once a second.

MoWitness Github Repo

What About the Clients?

What happens to the clients when they want to RENEW and the DHCP server they are bound to disappears?

This illustration probably explains it better than words:

Kea 1.3.0 Site Failover DHCP REBIND