Skip to content

Gobgp Keepalived IPVS

High-Performance Active-Active Load Balancing: Building a Maglev L4 Tier with GoBGP, Keepalived, and IPVS

At massive scale, traditional round-robin or least-connection load balancing algorithms suffer during node failures or scale-out events. When a load balancer node or backend host is added or removed, traditional hashing algorithms remap almost all connections, causing massive session disruptions across the fleet.

Google solved this problem with Maglev, a consistent hashing algorithm designed to distribute traffic evenly while minimizing connection re-mappings during pool changes.

By combining IPVS (for kernel-level packet forwarding), Keepalived (for backend health monitoring), GoBGP (for ECMP route advertisement), and the Maglev hashing scheduler, you can build a resilient, carrier-grade Layer 4 active-active load balancing tier.

Why Maglev Consistent Hashing?

In a standard Layer 4 load balancer using basic source-IP or 5-tuple hashing, changing the number of backend servers alters the lookup table modulus. This causes existing TCP connections to land on different backends after a topology change, breaking active sessions.

Key Benefits of Maglev Hashing in IPVS

  • Minimal Disruption: When a backend server goes down or a new one is added, only a minimal fraction (\(1/N\)) of connections are re-mapped. All other connections maintain their original backend target.
  • Equal Distribution: Maglev generates a lookup table (permutation array) that guarantees an even distribution of connections across all healthy targets, even with non-uniform server weights.
  • High Throughput: Operating inside the Linux kernel via IPVS (mh scheduler), Maglev handles millions of packets per second with negligible CPU overhead.

Architecture Overview

This setup uses a modern active-active network design:

  1. Upstream Routers (ToR Switches): Receive BGP route advertisements from all load balancer nodes for a shared Virtual IP (VIP). They use Equal-Cost Multi-Pathing (ECMP) to split incoming traffic evenly across the load balancers.
  2. GoBGP: Runs as a control plane daemon on each load balancer node to advertise/withdraw the /32 VIP route to upstream switches based on service health.
  3. Keepalived: Performs continuous application health checks on backend real servers and updates the local IPVS kernel table dynamically.
  4. IPVS (mh scheduler): Configured with the Maglev hashing algorithm to forward incoming packets to backend servers with minimal session re-mapping.

Bind the Virtual IP to a Dummy Interface

To accept traffic for the advertised VIP on an active-active node without triggering VRRP IP collisions, bind the VIP to a local dummy interface.

# Create dummy interface
ip link add dummy0 type dummy
ip addr add 192.168.100.1/32 dev dummy0
ip link set dummy0 up

Configure Keepalived with IPVS Maglev Scheduler

In Keepalived, set the load balancing algorithm (lb_algo) to mh (Maglev Hashing).

global_defs {
    router_id LB_NODE_01
    enable_script_security
}

# Define the Virtual Server using Maglev Hashing (mh)
virtual_server 192.168.100.1 80 {
    delay_loop 3
    lb_algo mh               # Enables Maglev Consistent Hashing in IPVS
    lb_kind NAT              # Or TUN/DR depending on your network topology
    protocol TCP

    # Real Backend Server 1
    real_server 10.0.1.11 80 {
        weight 100
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            retry 3
            delay_before_retry 2
        }
    }

    # Real Backend Server 2
    real_server 10.0.1.12 80 {
        weight 100
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            retry 3
            delay_before_retry 2
        }
    }

    # Real Backend Server 3
    real_server 10.0.1.13 80 {
        weight 100
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            retry 3
            delay_before_retry 2
        }
    }
}

Configure GoBGP for ECMP Route Advertisement

Configure GoBGP to establish a BGP session with your upstream router and advertise the VIP route (192.168.100.1/32).

[global.config]
  as = 65001
  router-id = "10.0.0.1"

[[neighbors]]
  [neighbors.config]
    neighbor-address = "10.0.0.254" # Upstream Switch/Router IP
    peer-as = 65000

[[neighbors.afi-safis]]
  [neighbors.afi-safis.config]
    afi-safi-name = "ipv4-unicast"
Start the GoBGP daemon and register the route 
# Start GoBGP daemon
gobgpd -f /etc/gobgp/gobgpd.conf &

# Advertise the VIP to the network
gobgp global rib add 192.168.100.1/32

Health Checking & Dynamic Route Pullback

If all backend servers behind a load balancer node fail, Keepalived flushes the IPVS table. To prevent the upstream switch from continuing to forward ECMP traffic to this empty node, use a health-check monitor script to withdraw the BGP route.

#!/bin/bash
# /usr/local/bin/bgp_healthcheck.sh

VIP="192.168.100.1/32"
VIP_PORT="192.168.100.1:80"

# Fetch IPVS table state
IPVS_OUTPUT=$(ipvsadm -Ln)

# Check if there are active backends for this VIP
ACTIVE_BACKENDS=$(echo "$IPVS_OUTPUT" | grep -A 10 "$VIP_PORT" | grep -c "Masq")

# Check if GoBGP is currently advertising the route
IS_ADVERTISED=$(gobgp global rib -a ipv4 | grep -c "192.168.100.1/32")

if [ "$ACTIVE_BACKENDS" -gt 0 ]; then
    # Backends are healthy: Ensure BGP route is advertised
    if [ "$IS_ADVERTISED" -eq 0 ]; then
        logger -t bgp_healthcheck "Active backends detected for $VIP. Advertising route via GoBGP."
        gobgp global rib add $VIP -a ipv4
    fi
else
    # No healthy backends: Withdraw BGP route to prevent blackholing
    if [ "$IS_ADVERTISED" -gt 0 ]; then
        logger -t bgp_healthcheck "CRITICAL: Zero healthy backends for $VIP. Withdrawing route from GoBGP."
        gobgp global rib del $VIP -a ipv4
    fi
fi
Make the script executable and schedule it using a systemd timer or cron job to run every 2–3 seconds.

Verification and Operational Checks

Verify IPVS Maglev Scheduler

Run ipvsadm to ensure the kernel is using the mh (Maglev Hashing) scheduler:

ipvsadm -Ln
Verify BGP Advertisements

Check GoBGP to confirm the route is active and sent to peers: 

gobgp neighbor 10.0.0.254 adj-out

Simulate Backend Failure

Stop one backend server (e.g., 10.0.1.11). Observe Keepalived removing 10.0.1.11 from the IPVS pool. Thanks to Maglev consistent hashing, connections mapped to 10.0.1.12 and 10.0.1.13 remain uninterrupted on their existing nodes, preventing cluster-wide TCP session drops.