Skip to content

Loadbalancing Healthchecks

Load Balancing Health Checks: Core Concepts & Implementation Guide

In high-availability and distributed systems, load balancers act as the central dispatcher for incoming network traffic. However, a load balancer is only as effective as its ability to detect backend failures. Without continuous health monitoring, traffic will inevitably be routed to dead, degraded, or unresponsive servers, leading to service outages and poor user experience.

Health checks serve as the vital feedback mechanism that allows load balancers to dynamically adjust routing tables, removing unhealthy backend nodes from the active pool until they recover.

Healthcheck Basics

At its core, a health check is an automated, periodic probe sent from a load balancer (or monitoring daemon) to a backend service to verify its operational status.

Key Concepts & Parameters

  1. Interval (Check Frequency): How often the probe is sent to each backend server (e.g., every 5 seconds). A shorter interval detects failures faster but increases traffic and CPU overhead on backend systems.
  2. Timeout: The maximum amount of time the load balancer waits for a response before marking that specific attempt as failed (e.g., 2 seconds).
  3. Rise (Fall Thresholds):
  4. Fall (Unhealthy Threshold): The number of consecutive failed probes required before a node is marked as DOWN and removed from the active pool (e.g., 3 consecutive failures).
  5. Rise (Healthy Threshold): The number of consecutive successful probes required before a previously failed node is marked as UP and returned to rotation (e.g., 2 consecutive successes).
  6. Flapping Prevention: Requiring multiple consecutive successes/failures prevents "flapping"—a state where a server rapidly flips between healthy and unhealthy due to temporary network jitter, overloading the remaining servers.

Passive vs. Active Health Checks

  • Active Health Checks: The load balancer proactively sends out synthetic probes (pings, TCP connections, HTTP GET requests) at fixed intervals regardless of live user traffic.
  • Passive Health Checks: The load balancer monitors real user traffic passing through to backend nodes. If a backend returns network errors or HTTP 5xx error responses to actual requests beyond a set threshold, the load balancer temporarily ejects the node from the pool.

Layer 4 Health Checks

Layer 4 (Transport Layer) health checks evaluate server availability at the TCP or UDP network layer. They focus on connection establishment rather than application response content.

Mechanism & Characteristics

  • How it works: An L4 check attempts to open a TCP connection (SYN packet) to a specific port on the target backend (e.g., port 80, 443, or 3306). If the backend returns a SYN-ACK within the timeout window, the node is considered healthy, and the connection is immediately terminated (often via RST or FIN).
  • Advantages: Extremely low resource consumption, lightning-fast execution, and protocol-agnostic.
  • Disadvantages: L4 checks only confirm that the network socket is open and accepting connections. They cannot detect application-level deadlocks, database connection pool exhaustion, or internal server errors (such as HTTP 500 responses).

Keepalived-Based L4 Health Checks

keepalived is a widely used routing and high-availability daemon that relies on the Linux Virtual Server (LVS) framework and VRRP protocol. It excels at performing fast, low-overhead L4 health checks to manage IP virtual servers.

keepalived supports basic TCP and UDP checks directly inside its configuration (keepalived.conf).

Configuration Breakdown

In keepalived, health checks are defined inside a real_server block within a virtual_server definition:

  • TCP_CHECK: Probes a specific port on the real server to verify socket availability.
  • connect_port: The port number to test.
  • connect_timeout: Timeout limit in seconds for establishing the TCP connection.
  • delay_before_retry: Time interval between consecutive retry attempts.
  • retry: Number of failed connection attempts before marking the real server as down.

Example Scenario

If a web server daemon crashes but the operating system remains responsive, keepalived's TCP_CHECK detects that port 80 is no longer accepting connections, and it immediately removes the backend IP from the kernel's LVS routing table.

Layer 7 Health Checks

Layer 7 (Application Layer) health checks inspect the actual service layer payload, status codes, and application context (e.g., HTTP, HTTPS, gRPC, DNS).

Mechanism & Characteristics

  • How it works: The health checker opens a TCP connection, completes any required TLS handshake, sends a full protocol request (such as GET /health or HEAD /status), and parses the application response.
  • Status & Content Verification: A node is considered healthy only if:
  • The TCP connection succeeds.
  • The TLS handshake succeeds (if applicable).
  • The response status code matches expected values (e.g., 200 OK or 2xx/3xx range).
  • Optional: The response body contains a specific expected string (e.g., "status": "ALIVE").
  • Deep Health Probing: Application teams can configure an internal /health endpoint that queries downstream dependencies (e.g., database connection, cache status, local disk space) before returning an HTTP 200 code.
  • Trade-offs: Requires more CPU, memory, and network resources than L4 checks, especially when evaluating encrypted HTTPS traffic.

HAProxy-Based L7 Health Checks

HAProxy is a high-performance Layer 4 and Layer 7 load balancer renowned for its sophisticated health-checking capabilities and fine-grained HTTP inspection options.

In HAProxy, health checks are configured in the backend section of haproxy.cfg.

Key Configuration Options

  • option httpchk: Enables HTTP-based health checking instead of basic TCP connection tests. It can be extended to specify HTTP methods and paths (e.g., option httpchk GET /health).
  • http-check expect: Defines strict conditions that the HTTP response must meet to pass health evaluation.
  • Status checking: http-check expect status 200 or http-check expect rstatus ^2[0-9]{2}$ (regex matching 2xx range).
  • String checking: http-check expect string "ok" (verifies specific content in the response body).
  • check keyword: Appended to backend server lines to activate health probing on that specific host.
  • inter, fall, rise parameters:
  • inter: Sets the check interval (e.g., inter 3000ms).
  • fall: Sets the consecutive failure count (e.g., fall 3).
  • rise: Sets the consecutive recovery count (e.g., rise 2).

Example Execution Flow

  1. HAProxy sends an encrypted GET /health request to each backend web server every 3 seconds.
  2. The application executes internal checks (verifying database connectivity and disk read/write capability).
  3. If the backend returns HTTP 200 OK along with the string "status": "healthy", HAProxy keeps the server in active rotation.
  4. If a backend returns HTTP 503 Service Unavailable or times out 3 consecutive times, HAProxy automatically routes all incoming traffic away from that node.

Conclusion

A resilient load-balancing strategy relies on choosing the appropriate health-checking depth:

  • Use Layer 4 health checks (e.g., Keepalived) for ultra-fast, high-throughput network routing where low latency and resource minimalization are paramount.
  • Use Layer 7 health checks (e.g., HAProxy) for application-aware load balancing where ensuring backend functional integrity (database connectivity, API availability, and accurate response payloads) is critical to preventing silent service disruptions.