Architecture

ExaBGP Architecture

Deep-dive into ExaBGP's internal architecture and design

This document provides a comprehensive technical overview of how ExaBGP works internally, its design principles, component architecture, and code organization.

Overview

ExaBGP is a pure BGP protocol implementation designed for programmable network automation. Unlike traditional BGP implementations (BIRD, FRRouting, Cisco IOS), ExaBGP does NOT manipulate routing tables or forward packets.

What ExaBGP IS

✅ Complete BGP-4 protocol implementation (RFC 4271)
✅ Support for 55+ BGP RFCs and extensions
✅ API for dynamic route control
✅ Multiprotocol BGP (IPv4, IPv6, VPN, FlowSpec, EVPN, BGP-LS)
✅ Language-agnostic external process integration

What ExaBGP is NOT

❌ Does NOT maintain RIB (Routing Information Base)
❌ Does NOT manipulate FIB (Forwarding Information Base)
❌ Does NOT forward packets
❌ Does NOT run IGP protocols (OSPF, IS-IS)
❌ Does NOT modify kernel routing tables

Key Insight: ExaBGP speaks the BGP protocol. External processes decide what to announce. The OS kernel or other software performs forwarding.

Core Design Principles

1. Separation of Concerns

┌────────────────────────────────────────────────────┐
│  Business Logic (Health Checks, DDoS Detection)    │  ← Your Code
├────────────────────────────────────────────────────┤
│  BGP Protocol (Session Management, Messages)       │  ← ExaBGP
├────────────────────────────────────────────────────┤
│  Route Installation (ip route add, FRR, BIRD)      │  ← External Tools
├────────────────────────────────────────────────────┤
│  Packet Forwarding (Kernel, Hardware)              │  ← OS/Hardware
└────────────────────────────────────────────────────┘

Rationale:

Focus: ExaBGP focuses on BGP protocol correctness
Flexibility: Business logic in any language
Simplicity: No complex routing policy language
Security: Reduced attack surface (no kernel modifications)

2. API-First Architecture

Everything is controlled via STDIN/STDOUT pipes between ExaBGP and external processes.

Command Flow:

External Process → STDOUT → ExaBGP STDIN → BGP UPDATE → Neighbor Router

Message Flow:

Neighbor Router → BGP UPDATE → ExaBGP → Process STDIN → External Process

Advantages:

Language-agnostic (Python, Shell, Go, Rust, etc.)
Simple integration (just read/write text)
No complex SDKs or libraries required
Easy testing (pipe commands manually)

3. No RIB/FIB Manipulation

Traditional BGP Implementation:

BGP Peer → Receive Route → Update RIB → Select Best → Install in FIB → Forward Packets

ExaBGP:

BGP Peer → Receive Route → Send to External Process → [External Process Decides]

Why?

Flexibility: External process can use any routing logic
Simplicity: No complex route selection algorithm
Integration: Easy to integrate with existing routing systems
Separation: BGP protocol separate from routing policy

Implication: If you want routes installed in the kernel, you must write code to call ip route add.

See also: RFC Support - No RIB/FIB Manipulation

High-Level Architecture

System Overview

┌──────────────────────────────────────────────────────────────────┐
│                      External Processes                          │
│                                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐            │
│  │ Health Check │  │ DDoS Detect  │  │ Custom Logic │            │
│  │   (Python)   │  │   (Shell)    │  │    (Go)      │            │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘            │
│         │ STDOUT          │ STDOUT          │ STDOUT             │
│         │ Commands        │ Commands        │ Commands           │
└─────────┼─────────────────┼─────────────────┼────────────────────┘
          ▼                 ▼                 ▼
┌──────────────────────────────────────────────────────────────────┐
│                          ExaBGP Core                             │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │                    Process Manager                         │  │
│  │  • Launch/monitor external processes                       │  │
│  │  • Manage STDIN/STDOUT pipes                               │  │
│  │  • Restart crashed processes                               │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌────────────────┐  ┌──────────────┐  ┌────────────────────┐    │
│  │     Parser     │  │   Encoder    │  │   Configuration    │    │
│  │  (Commands)    │  │ (JSON/Text)  │  │     Parser         │    │
│  └────────────────┘  └──────────────┘  └────────────────────┘    │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │                  BGP Protocol Engine                       │  │
│  │                                                            │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │  │
│  │  │   Neighbor   │  │   Message    │  │   Capability     │  │  │
│  │  │  Management  │  │   Builder    │  │   Negotiation    │  │  │
│  │  └──────────────┘  └──────────────┘  └──────────────────┘  │  │
│  │                                                            │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │  │
│  │  │   FSM        │  │   Timer      │  │   Attribute      │  │  │
│  │  │ (RFC 4271)   │  │  Management  │  │   Handling       │  │  │
│  │  └──────────────┘  └──────────────┘  └──────────────────┘  │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
└─────────────┬────────────────────────────────────────────────────┘
              │ TCP Port 179
              │ BGP Protocol
              ▼
┌──────────────────────────────────────────────────────────────────┐
│                      BGP Neighbor (Router)                       │
│         (Cisco, Juniper, Arista, FRRouting, BIRD, etc.)          │
└──────────────────────────────────────────────────────────────────┘

Component Architecture

Process Manager

Purpose: Manage external processes that control ExaBGP via API.

Responsibilities:

Launch processes - Fork configured processes at startup
Monitor health - Detect crashed processes
Auto-restart - Restart failed processes (configurable)
Pipe management - Establish STDIN/STDOUT pipes
Clean shutdown - Terminate processes on ExaBGP shutdown

Configuration Example:

process healthcheck {
    run python3 -m exabgp healthcheck --cmd "curl http://localhost";
    encoder text;
}

process ddos-detect {
    run /etc/exabgp/scripts/ddos-detect.py;
    encoder json;
}

Process Lifecycle:

ExaBGP Startup
    ↓
Read Configuration
    ↓
For Each Configured Process:
    ↓
    Fork Process → Create Pipes → Monitor
    ↓
Process Running ←──┐
    ↓              │
Process Crash?     │
    ↓ Yes          │
Auto-restart ──────┘
    ↓ No
Continue

See also: Process Configuration

Command Parser

Purpose: Parse commands from external processes and execute them.

Input: STDOUT from external processes Output: Internal command queue

Supported Formats:

Text API - Human-readable text commands
JSON API - Structured JSON commands

Text Command Example:

announce route 10.0.0.0/24 next-hop 192.0.2.1 community [65001:100]
withdraw route 10.0.0.0/24
announce flow route { match { source 10.0.0.0/8; } then { discard; } }
shutdown

JSON Command Example:

{
  "exabgp": "4.0",
  "type": "announce",
  "route": {
    "nlri": "10.0.0.0/24",
    "next-hop": "192.0.2.1",
    "community": [[65001, 100]]
  }
}

Parsing Flow:

External Process STDOUT
    ↓
Read Line
    ↓
Detect Format (text vs json)
    ↓
Parse Command
    ↓
Validate Syntax
    ↓
Queue for Execution
    ↓
Execute Command
    ↓
Send ACK (5.x only)

See also: Text API Reference, JSON API Reference

Encoder

Purpose: Format received BGP messages for external processes.

Input: BGP messages from neighbors Output: Formatted messages to process STDIN

Text Format:

neighbor 192.0.2.1 192.0.2.2 received update route 10.0.0.0/24 \
  next-hop 192.0.2.1 origin igp as-path [65001 65002]

JSON Format:

{
  "exabgp": "4.0.0",
  "time": 1699999999.0,
  "type": "update",
  "neighbor": {
    "address": {"local": "192.0.2.2", "peer": "192.0.2.1"},
    "asn": {"local": 65000, "peer": 65001}
  },
  "message": {
    "update": {
      "announce": {
        "ipv4 unicast": {
          "192.0.2.1": [{
            "nlri": "10.0.0.0/24",
            "attribute": {
              "origin": "igp",
              "as-path": [65001, 65002],
              "next-hop": "192.0.2.1"
            }
          }]
        }
      }
    }
  }
}

Message Types Encoded:

update - Route announcements/withdrawals
open - BGP OPEN message received
notification - BGP NOTIFICATION (errors)
keepalive - BGP KEEPALIVE
refresh - Route refresh request
state - BGP session state changes
connected / down - Session events

See also: API Overview

Configuration Parser

Purpose: Parse ExaBGP configuration file.

Syntax: INI-style with hierarchical blocks

Example:

[exabgp.api]
ack = true

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    family {
        ipv4 unicast;
        ipv4 flow;
    }

    static {
        route 10.0.0.0/24 next-hop 192.0.2.1;
    }

    api {
        processes [ healthcheck ];
    }
}

process healthcheck {
    run python3 -m exabgp healthcheck;
    encoder text;
}

Parsing Features:

Template inheritance
Include files
Environment variable expansion
Syntax validation

See also: Configuration Syntax

BGP Protocol Implementation

BGP Finite State Machine (FSM)

RFC 4271 - ExaBGP implements the complete BGP-4 FSM.

States:

┌──────┐
│ Idle │ ← Initial state, session not established
└───┬──┘
    │ Start event (configured neighbor)
    ▼
┌─────────┐
│ Connect │ ← Initiating TCP connection
└────┬────┘
     │ TCP connection established
     ▼
┌──────────┐
│OpenSent  │ ← OPEN message sent, waiting for peer OPEN
└────┬─────┘
     │ Received valid OPEN message
     ▼
┌─────────────┐
│OpenConfirm  │ ← OPEN confirmed, waiting for KEEPALIVE
└──────┬──────┘
       │ Received KEEPALIVE
       ▼
┌─────────────┐
│Established  │ ← Session established, exchanging routes
└─────────────┘

State Transitions:

Idle → Connect - Configuration loaded
Connect → OpenSent - TCP connection established
OpenSent → OpenConfirm - Valid OPEN received
OpenConfirm → Established - KEEPALIVE received
Established → Idle - Error, NOTIFICATION, or manual shutdown

See also: BGP State Machine

BGP Message Types

ExaBGP implements all BGP-4 message types:

1. OPEN Message

Purpose: Session establishment and capability negotiation

Fields:

Version (always 4)
AS Number (16-bit or 32-bit)
Hold Time
BGP Identifier (Router ID)
Optional Parameters (capabilities)

Capabilities Negotiated:

Multiprotocol Extensions (IPv6, VPN, FlowSpec, etc.)
4-byte ASN Support
ADD-PATH
Graceful Restart
Route Refresh
Extended Message Support

See also: Capabilities

2. UPDATE Message

Purpose: Route announcements and withdrawals

Structure:

┌────────────────────────────────────┐
│ Withdrawn Routes Length (2 bytes)  │
├────────────────────────────────────┤
│ Withdrawn Routes (variable)        │
├────────────────────────────────────┤
│ Path Attributes Length (2 bytes)   │
├────────────────────────────────────┤
│ Path Attributes (variable)         │
│  - ORIGIN                          │
│  - AS_PATH                         │
│  - NEXT_HOP                        │
│  - LOCAL_PREF                      │
│  - COMMUNITY                       │
│  - etc.                            │
├────────────────────────────────────┤
│ NLRI (Network Layer Reachability)  │
│  - Prefix length + IP prefix       │
└────────────────────────────────────┘

Multiprotocol (MP-BGP): For non-IPv4 unicast, routes are carried in MP_REACH_NLRI and MP_UNREACH_NLRI attributes.

See also: Attribute Reference

3. NOTIFICATION Message

Purpose: Error reporting and session termination

Error Codes:

Message Header Error
OPEN Message Error
UPDATE Message Error
Hold Timer Expired
Finite State Machine Error
Cease

Subcodes: Specific error within each code

See also: Debugging Guide

4. KEEPALIVE Message

Purpose: Session maintenance

Frequency: Sent every (hold-time / 3) seconds

Format: Empty message (header only)

5. ROUTE-REFRESH Message

Purpose: Request re-advertisement of routes (RFC 2918)

Use Case: Policy changes without session reset

Message Builder

Purpose: Construct BGP UPDATE messages from API commands.

Flow:

API Command
    ↓
Parse Attributes
    ↓
Validate Syntax
    ↓
Build BGP Attributes
    ↓
Construct UPDATE Message
    ↓
Send to Neighbor

Attribute Encoding:

Type-Length-Value (TLV) format
Flags: Optional/Well-Known, Transitive/Non-Transitive, Partial, Extended
Correct byte ordering (network byte order)

Address Family Support:

IPv4 Unicast (traditional NLRI)
IPv6, VPN, FlowSpec, EVPN, BGP-LS (MP_REACH_NLRI)

See also: Address Families

Neighbor Management

Purpose: Manage BGP sessions with configured neighbors.

Per-Neighbor State:

TCP connection
BGP FSM state
Hold timer / Keepalive timer
Capabilities negotiated
Session statistics

Capabilities Negotiation:

Local Capabilities ∩ Peer Capabilities = Negotiated Capabilities

Timer Management:

Hold Timer: Countdown from hold-time, reset on UPDATE/KEEPALIVE
Keepalive Timer: Trigger KEEPALIVE every (hold-time / 3)

Capability Negotiation

Purpose: Agree on optional BGP features during OPEN exchange.

Common Capabilities:

Multiprotocol Extensions (RFC 4760) - AFI/SAFI support
4-byte ASN (RFC 4893) - 32-bit AS numbers
ADD-PATH (RFC 7911) - Multiple paths per prefix
Graceful Restart (RFC 4724) - Forwarding preservation
Route Refresh (RFC 2918) - Route re-advertisement
Extended Message (RFC 8654) - Messages > 4096 bytes

Negotiation Process:

Local ExaBGP advertises capabilities in OPEN
Peer advertises capabilities in OPEN
Intersection of capabilities is used
Unsupported capabilities are ignored

See also: Capabilities, RFC Compliance

Process and API Architecture

API Communication Model

Bi-directional pipes:

┌────────────────┐
│ External       │
│ Process        │
│                │
│ stdin  stdout  │
└───▲────────┬───┘
    │        │
    │        │ Commands
    │        ▼
┌───┴────────▼───┐
│   ExaBGP       │
│                │
│  BGP Messages  │
└────────────────┘

External Process Perspective:

Read from STDIN - Receive BGP messages from ExaBGP
Write to STDOUT - Send commands to ExaBGP
Flush after write - Critical for real-time communication

ExaBGP Perspective:

Read from Process STDOUT - Receive commands
Write to Process STDIN - Send BGP messages

Command Acknowledgment (ACK)

ExaBGP 4.x and 5.x feature (enabled by default) - Sends ACK after executing commands.

Responses:

done - Command executed successfully
error <message> - Command failed with error
shutdown - ExaBGP is shutting down

Example Flow:

import sys
import select
import time

def wait_for_ack(expected_count=1, timeout=30):
    """Wait for ACK with polling loop. Handles text and JSON formats."""
    import json
    received = 0
    start_time = time.time()

    while received < expected_count:
        if time.time() - start_time >= timeout:
            return False

        ready, _, _ = select.select([sys.stdin], [], [], 0.1)
        if ready:
            line = sys.stdin.readline().strip()

            # Parse response (could be text or JSON)
            answer = None
            if line.startswith('{'):
                try:
                    data = json.loads(line)
                    answer = data.get('answer')
                except:
                    pass
            else:
                answer = line

            if answer == "done":
                received += 1
            elif answer == "error":
                return False
            elif answer == "shutdown":
                raise SystemExit(0)
        else:
            time.sleep(0.1)

    return True

# Send command
sys.stdout.write("announce route 10.0.0.0/24 next-hop 192.0.2.1\n")
sys.stdout.flush()

# Wait for ACK (with polling)
if not wait_for_ack():
    sys.exit(1)  # Command failed

Available in both 4.x and 5.x (enabled by default).

See also: API Overview - ACK Feature

Event Loop Architecture

ExaBGP uses a polling-based event loop (reactor pattern):

┌────────────────────────────────┐
│        Event Loop              │
│                                │
│  ┌──────────────────────────┐  │
│  │ Loop with Small Sleep    │  │
│  │  - Select() for I/O      │  │
│  │    • TCP sockets         │  │
│  │    • Process pipes       │  │
│  │  - Check Timers          │  │
│  │    • Hold timer          │  │
│  │    • Keepalive timer     │  │
│  └──────────────────────────┘  │
│                                │
│  Loop Iteration:               │
│    ├─ Select(timeout) for I/O  │
│    ├─ TCP Data? → Process BGP  │
│    ├─ Process Data? → Parse    │
│    ├─ Check Timers → Action    │
│    ├─ Sleep briefly            │
│    └─ Repeat                   │
└────────────────────────────────┘

Implementation Details:

I/O: Uses select() for non-blocking socket/pipe I/O
Timers: Implemented as loop-based polling - not event-driven
- Each iteration checks if timers have expired
- Small sleep between iterations (reduces CPU usage)
- Not using timer file descriptors (timerfd) or true event-driven timers
Single-threaded: Simplicity, no locking required
Non-blocking I/O: Efficient resource usage

Why Polling for Timers?

Simplicity: No platform-specific timer APIs
Portability: Works consistently across OS platforms
Sufficient: BGP timers are typically seconds/minutes (not milliseconds)

Code Organization

Directory Structure

Main codebase location: /src/exabgp/

src/exabgp/
├── application/           # ExaBGP main application entry point
├── bgp/                   # BGP protocol implementation
│   ├── message/          # BGP message types (OPEN, UPDATE, etc.)
│   │   ├── open/         # OPEN message and capabilities
│   │   ├── update/       # UPDATE message and attributes
│   │   ├── notification/ # NOTIFICATION messages
│   │   └── refresh.py    # ROUTE-REFRESH
│   ├── neighbor.py       # Neighbor management
│   └── fsm.py            # Finite State Machine
├── configuration/         # Configuration parsing
│   ├── neighbor/         # Neighbor configuration
│   ├── family/           # Address family configuration
│   └── process/          # Process configuration
├── protocol/              # Protocol utilities
│   ├── ip/               # IP address handling
│   ├── family.py         # AFI/SAFI definitions
│   └── attributes.py     # BGP attribute handling
├── reactor/               # Event loop (reactor pattern)
│   ├── loop.py           # Main event loop
│   ├── api/              # API (process communication)
│   └── network/          # Network I/O
├── rib/                   # Adj-RIB-In/Out (limited use)
└── version.py             # Version information

Key Modules:

Module	Purpose
`application/`	Main entry point, CLI
`bgp/message/`	BGP message encoding/decoding
`bgp/neighbor.py`	Neighbor state management
`configuration/`	Config file parsing
`reactor/`	Event loop and I/O
`protocol/`	Protocol utilities

Key Classes

Neighbor

File: src/exabgp/bgp/neighbor.py

Responsibilities:

Manage BGP session with one peer
Track FSM state
Store negotiated capabilities
Maintain timers

Key Methods:

connect() - Initiate TCP connection
open() - Send OPEN message
update() - Send UPDATE message
keepalive() - Send KEEPALIVE

Message

File: src/exabgp/bgp/message/

Hierarchy:

Message (abstract)
├── Open
├── Update
│   ├── Announce
│   └── Withdraw
├── Notification
├── KeepAlive
└── RouteRefresh

Responsibilities:

Encode BGP messages to wire format
Decode received BGP messages
Validate message correctness

Attribute

File: src/exabgp/bgp/message/update/attribute/

Supported Attributes:

ORIGIN, AS_PATH, NEXT_HOP
LOCAL_PREF, MED
COMMUNITY, EXTENDED_COMMUNITY, LARGE_COMMUNITY
MP_REACH_NLRI, MP_UNREACH_NLRI
ORIGINATOR_ID, CLUSTER_LIST
And 20+ more

See also: Attribute Reference

Reactor (Event Loop)

File: src/exabgp/reactor/loop.py

Responsibilities:

Event-driven I/O multiplexing
Process BGP messages from neighbors
Process commands from external processes
Manage timers
Handle signals (reload, shutdown)

Extension Points

ExaBGP is designed to be extended. Here are the main extension points:

1. Custom Address Families

Add support for new AFI/SAFI combinations:

Define AFI/SAFI in src/exabgp/protocol/family.py
Implement NLRI encoding/decoding
Add configuration parser support
Add API command support

Example: SRv6 Mobile User Plane (MUP) was added this way.

2. Custom Attributes

Add support for new BGP attributes:

Implement attribute class in src/exabgp/bgp/message/update/attribute/
Add encoding/decoding logic
Add to attribute registry
Update configuration parser

See also: Generic Attribute Support

3. Custom Process Encoders

Create new API formats beyond text/json:

Implement encoder in src/exabgp/reactor/api/encoder/
Register encoder in configuration
Handle message formatting

4. Plugins

ExaBGP supports plugins for extending functionality without modifying core code.

Plugin use cases:

Custom health check logic
Specialized message processing
Integration with external systems

Performance Considerations

Single-Threaded Event Loop

Design: ExaBGP uses a single-threaded event loop (reactor pattern).

Implications:

✅ Simple, no locking required
✅ Predictable performance
❌ CPU-bound operations block event loop
❌ Limited to single CPU core

Best Practice: Keep external process logic fast. Use separate processes for CPU-intensive tasks.

Memory Efficiency

ExaBGP does NOT store full routing tables:

No RIB/FIB → Very low memory footprint
Only tracks: configured static routes, active sessions
Typical memory: < 100 MB

Contrast: Traditional BGP implementations store millions of routes (GB of RAM).

Scalability

Neighbor scaling:

Tested with 100+ BGP neighbors
Limited by event loop (single thread)

Route scaling:

No limit on announced routes (external process decides)
UPDATE message rate limited by neighbor capacity

Process scaling:

Multiple external processes supported
Each process runs independently

Version Differences

ExaBGP 3.x (Legacy)

Characteristics:

Python 2 only
Limited address family support
Text API only
No JSON encoder

Status: Deprecated, no longer maintained

ExaBGP 4.x (Stable)

Characteristics:

Python 3 support
Full multiprotocol support (IPv6, VPN, FlowSpec, EVPN, BGP-LS)
JSON API
Wide adoption

API: No command acknowledgment

Configuration: Environment variables + config file

See also: Migration from 3.4 to 4.x

ExaBGP 5.x / main (Latest)

New Features:

Command acknowledgment (ACK) - done, error, shutdown responses
flush route command - Withdraw all routes
Enhanced JSON API format
Improved error messages

Breaking changes:

ACK changes API behavior (must handle responses)
Some configuration syntax updates

See also: Migration from 4.2 to 5.0, API Overview - ACK

Architecture

ExaBGP Architecture

Table of Contents

Overview

What ExaBGP IS

What ExaBGP is NOT

Core Design Principles

1. Separation of Concerns

2. API-First Architecture

3. No RIB/FIB Manipulation

High-Level Architecture

System Overview

Component Architecture

Process Manager

Command Parser

Encoder

Configuration Parser

BGP Protocol Implementation

BGP Finite State Machine (FSM)

BGP Message Types

1. OPEN Message

2. UPDATE Message

3. NOTIFICATION Message

4. KEEPALIVE Message

5. ROUTE-REFRESH Message

Message Builder

Neighbor Management

Capability Negotiation

Process and API Architecture

API Communication Model

Command Acknowledgment (ACK)

Event Loop Architecture

Code Organization

Directory Structure

Key Classes

Neighbor

Message

Attribute

Reactor (Event Loop)

Extension Points

1. Custom Address Families

2. Custom Attributes

3. Custom Process Encoders

4. Plugins

Performance Considerations

Single-Threaded Event Loop

Memory Efficiency

Scalability

Version Differences

ExaBGP 3.x (Legacy)

ExaBGP 4.x (Stable)

ExaBGP 5.x / main (Latest)

See Also

Documentation

Implementation Details

Use Cases

Development

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!