Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Summary:

- **`config/system/`** — Creates the `galactic-system` namespace both components deploy into. Apply with `kubectl apply -k config/system/`.
- **`config/cni/`** — Production manifests for the CNI installer DaemonSet, ConfigMap, RBAC, and ServiceAccount. Apply with `kubectl apply -k config/cni/`.
- **`config/router/`** — Shared RBAC/ServiceAccount plus DaemonSet roles, all running `GALACTIC_ROUTER_ROUTER_MODE=tenant`:
- **`config/router/`** — Shared RBAC/ServiceAccount plus DaemonSet roles:
- **`config/router/tenant/`** — the per-node role (`galactic-router`); runs on every node except Kubernetes control-plane nodes and nodes labeled for the route-reflector or gateway roles.
- **`config/router/tenant-control/`** — the BGP route-reflector role (`galactic-router-control`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires nodes labeled `galactic.datumapis.com/node: control` (stays at zero replicas otherwise). `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it.
- **`config/router/base/`** — the DaemonSet spec shared by both roles above; not applied directly.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,9 @@ kubectl apply -k config/fabric/

- **Talos: gRPC health port.** `galactic-router` runs `hostNetwork: true` and defaults to gRPC health checks on port `5000`, which collides with Talos's built-in dashboard (`/sbin/dashboard` permanently binds `127.0.0.1:5000` on every Talos node). `config/router/base/daemonset.yaml` already ships with `GALACTIC_ROUTER_GRPC_HEALTH_PORT=5179` (and matching probe/containerPort) to avoid this; if you run `galactic-router` outside these manifests on Talos, set `GALACTIC_ROUTER_GRPC_HEALTH_PORT` to something other than `5000` yourself.

- **`galactic-router` tenant mode: BGP local address.** The node needs a global-unicast IPv6 address assigned to `lo` (typically by `config/fabric/`'s underlay eBGP daemon, which must start and converge before `galactic-router`), or you must set `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` explicitly — this is required even when `GALACTIC_ROUTER_BGP_LISTEN_PORT=-1` (no inbound listener), since `galactic-router` still needs a source address for outbound BGP connections. Without one of these, startup fails with `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS not set and no address could be detected on lo: no global-unicast IPv6 address found on lo`. See [`docs/router/configuration.md`](./docs/router/configuration.md) for details.
- **`galactic-router`: BGP local address.** The node needs a global-unicast IPv6 address assigned to `lo` (typically by `config/fabric/`'s underlay eBGP daemon, which must start and converge before `galactic-router`), or you must set `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` explicitly — this is required even when `GALACTIC_ROUTER_BGP_LISTEN_PORT=-1` (no inbound listener), since `galactic-router` still needs a source address for outbound BGP connections. Without one of these, startup fails with `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS not set and no address could be detected on lo: no global-unicast IPv6 address found on lo`. See [`docs/router/configuration.md`](./docs/router/configuration.md) for details.

See [`docs/router/configuration.md`](./docs/router/configuration.md) for the full `galactic-router` CLI flag / environment variable reference — note that env var names generally follow `GALACTIC_ROUTER_<FLAG_NAME>` but aren't always the naive uppercased guess (e.g. `--mode` is `GALACTIC_ROUTER_ROUTER_MODE`, not `GALACTIC_ROUTER_MODE`); the reference table has the exact name for every flag.
See [`docs/router/configuration.md`](./docs/router/configuration.md) for the full `galactic-router` CLI flag / environment variable reference — env var names follow `GALACTIC_ROUTER_<FLAG_NAME>` (hyphens become underscores, uppercased); the reference table has the exact name for every flag.

## Development

Expand Down
3 changes: 1 addition & 2 deletions cmd/galactic-router/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@
// SPDX-License-Identifier: AGPL-3.0-or-later

// Command galactic-router is the BGP control-plane reconciler for the Galactic
// data plane. It watches BGP CRDs and drives a BGP runtime backend
// (GoBGP for tenant role, FRR stub for fabric role).
// data plane. It watches BGP CRDs and drives an embedded GoBGP server.
package main

import (
Expand Down
20 changes: 3 additions & 17 deletions cmd/galactic-router/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ import (
"go.datum.net/galactic/internal/plumbing/loaddr"
"go.datum.net/galactic/internal/reconcile"
galacticruntime "go.datum.net/galactic/internal/runtime"
"go.datum.net/galactic/internal/runtime/frr"
"go.datum.net/galactic/internal/runtime/gobgp"
networkwebhook "go.datum.net/galactic/internal/webhook"
bgpv1alpha1 "go.datum.net/network/api/v1alpha1"
Expand Down Expand Up @@ -66,7 +65,6 @@ func resolveBGPLocalAddress(explicit string, detect func() (string, error)) (str
// the provided config and initializes the BGP runtime.
func runCmd(cfg *config.RouterConfig) error {
nodeName := cfg.NodeName
mode := cfg.Mode
bgpListenPort := cfg.BGPListenPort
metricsPort := cfg.MetricsPort
grpcHealthPort := cfg.GRPCHealthPort
Expand All @@ -76,15 +74,7 @@ func runCmd(cfg *config.RouterConfig) error {
return err
}

var factory galacticruntime.RuntimeFactory
switch mode {
case config.ModeTenant:
factory = gobgp.NewRuntimeFactory(int32(bgpListenPort), bgpLocalAddr)
case config.ModeFabric:
factory = frr.NewRuntimeFactory()
case config.ModeTransit:
return errors.New("mode=transit is not yet supported")
}
factory := gobgp.NewRuntimeFactory(int32(bgpListenPort), bgpLocalAddr)

ctrl.SetLogger(zap.New(zap.UseDevMode(true)))

Expand Down Expand Up @@ -174,7 +164,7 @@ func runCmd(cfg *config.RouterConfig) error {
runtimeMgr := galacticruntime.NewRuntimeManager(factory)

// Create reconciler.
rec := reconcile.New(mgr.GetClient(), nodeName, mode, bgpLocalAddr)
rec := reconcile.New(mgr.GetClient(), nodeName, bgpLocalAddr)

// Register BGPRouter controller.
if err := (&controller.BGPRouterReconciler{
Expand All @@ -184,7 +174,6 @@ func runCmd(cfg *config.RouterConfig) error {
RuntimeManager: runtimeMgr,
Hasher: hash.DesiredRouter,
NodeName: nodeName,
RouterMode: mode,
}).SetupWithManager(mgr); err != nil {
return fmt.Errorf("setup BGPRouter controller: %w", err)
}
Expand Down Expand Up @@ -333,10 +322,7 @@ func newRootCommand() *cobra.Command {
}

cmd.Flags().StringP("node-name", "n", "", "Kubernetes node name (required)")
cmd.Flags().StringP("mode", "m", "",
"Operating mode: '"+config.ModeTransit+"', '"+config.ModeFabric+"', or '"+config.ModeTenant+"' (required)")
cmd.Flags().Bool("reflector", false,
"Enable route reflector mode (requires --mode="+config.ModeFabric+" or --mode="+config.ModeTenant+")")
cmd.Flags().Bool("reflector", false, "Enable route reflector mode")
cmd.Flags().IntP("bgp-listen-port", "p", config.DefaultRouterBGPListenPort,
"BGP listen port")
cmd.Flags().StringP("bgp-local-address", "",
Expand Down
85 changes: 1 addition & 84 deletions cmd/galactic-router/root_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ func testCmd(t *testing.T) *cobra.Command {
t.Helper()
cmd := &cobra.Command{Use: testCmdUse}
cmd.Flags().StringP("node-name", "n", "", "Kubernetes node name (required)")
cmd.Flags().StringP("mode", "m", "", "Operating mode")
cmd.Flags().Bool("reflector", false, "Enable route reflector mode")
cmd.Flags().IntP("bgp-listen-port", "p", config.DefaultRouterBGPListenPort, "BGP listen port")
cmd.Flags().StringP("bgp-local-address", "", "", "BGP local address")
Expand Down Expand Up @@ -56,13 +55,12 @@ func TestRequiredFlags(t *testing.T) {
cmd := testCmd(t)
cfg.BindFlags(cmd.Flags())
if err := cfg.Validate(); err == nil {
t.Error("Validate() with empty node-name and mode returned nil error")
t.Error("Validate() with empty node-name returned nil error")
}
}

func TestEnvVarDefaults(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)

cfg := config.NewRouterConfig()
cmd := testCmd(t)
Expand All @@ -72,21 +70,8 @@ func TestEnvVarDefaults(t *testing.T) {
}
}

func TestInvalidMode(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, "invalid")

cfg := config.NewRouterConfig()
cmd := testCmd(t)
cfg.BindFlags(cmd.Flags())
if err := cfg.Validate(); err == nil {
t.Error("Validate() with invalid mode returned nil error")
}
}

func TestBGPListenPortMinusOne(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)
t.Setenv(config.EnvRouterBGPListenPort, "-1")

cfg := config.NewRouterConfig()
Expand All @@ -99,7 +84,6 @@ func TestBGPListenPortMinusOne(t *testing.T) {

func TestBGPListenPortOverflow(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)
t.Setenv(config.EnvRouterBGPListenPort, "70000")

cfg := config.NewRouterConfig()
Expand All @@ -119,56 +103,8 @@ func TestNodeNameRequired(t *testing.T) {
}
}

func TestModeRequired(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")

cfg := config.NewRouterConfig()
cmd := testCmd(t)
cfg.BindFlags(cmd.Flags())
if err := cfg.Validate(); err == nil {
t.Error("Validate() with empty mode returned nil error")
}
}

func TestValidModes(t *testing.T) {
for _, mode := range []string{config.ModeTransit, config.ModeFabric, config.ModeTenant} {
t.Run(mode, func(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, mode)

cfg := config.NewRouterConfig()
cmd := testCmd(t)
cfg.BindFlags(cmd.Flags())

if cfg.Mode != mode {
t.Errorf("Mode = %q, want %q", cfg.Mode, mode)
}
if err := cfg.Validate(); err != nil {
t.Errorf("Validate() with mode %q: %v", mode, err)
}
})
}
}

func TestReflectorInvalidMode(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTransit)

cfg := config.NewRouterConfig()
cmd := testCmd(t)
if err := cmd.Flags().Set("reflector", "true"); err != nil {
t.Fatalf("set --reflector flag: %v", err)
}
cfg.BindFlags(cmd.Flags())

if err := cfg.Validate(); err == nil {
t.Error("Validate() with --reflector and --mode=transit returned nil error")
}
}

func TestMetricsPortOverride(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)
t.Setenv(config.EnvRouterMetricsPort, "9090")

cfg := config.NewRouterConfig()
Expand All @@ -181,7 +117,6 @@ func TestMetricsPortOverride(t *testing.T) {

func TestGRPCHealthPortOverride(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)
t.Setenv(config.EnvRouterGRPCHealthPort, "9091")

cfg := config.NewRouterConfig()
Expand All @@ -192,25 +127,8 @@ func TestGRPCHealthPortOverride(t *testing.T) {
}
}

func TestModeFlagOverridesEnv(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeFabric)

cfg := config.NewRouterConfig()
cmd := testCmd(t)
if err := cmd.Flags().Set("mode", config.ModeTenant); err != nil {
t.Fatalf("set --mode flag: %v", err)
}
cfg.BindFlags(cmd.Flags())

if cfg.Mode != config.ModeTenant {
t.Errorf("Mode = %q, want %q (flag should override env var)", cfg.Mode, config.ModeTenant)
}
}

func TestGRPCHealthPortFlagOverridesEnv(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)
t.Setenv(config.EnvRouterGRPCHealthPort, "9091")

cfg := config.NewRouterConfig()
Expand All @@ -227,7 +145,6 @@ func TestGRPCHealthPortFlagOverridesEnv(t *testing.T) {

func TestWebhookFlagsOverrideEnv(t *testing.T) {
t.Setenv(config.EnvRouterNodeName, "test-node")
t.Setenv(config.EnvRouterMode, config.ModeTenant)
t.Setenv(config.EnvRouterWebhookEnabled, "false")
t.Setenv(config.EnvRouterWebhookPort, "9443")

Expand Down
2 changes: 0 additions & 2 deletions config/gateway/base/daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,6 @@ spec:
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: GALACTIC_ROUTER_ROUTER_MODE
value: tenant
- name: GALACTIC_ROUTER_GC_NAMESPACE
value: galactic-system
- name: GALACTIC_ROUTER_BGP_LISTEN_PORT
Expand Down
2 changes: 0 additions & 2 deletions config/router/base/daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,6 @@ spec:
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: GALACTIC_ROUTER_ROUTER_MODE
value: tenant
- name: GALACTIC_ROUTER_GC_NAMESPACE
value: galactic-system
# 5179 is also the binary's own default. Set explicitly here
Expand Down
8 changes: 4 additions & 4 deletions docs/agent-startup.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ sequenceDiagram
participant GoBGP
participant Kubernetes

Router->>Router: validate config (--node-name, --mode: transit/fabric/tenant, ...)
Router->>Router: select RuntimeFactory: tenant→GoBGP, fabric→FRR stub, transit→error (unsupported)
Router->>Router: validate config (--node-name, ...)
Router->>Router: create RuntimeFactory (GoBGP — the only backend)
Router->>Kubernetes: build controller-runtime manager (metrics on :9179, no HTTP health)
Router->>Router: start gRPC health server (:5179, SERVING immediately)
Router->>Kubernetes: RBAC pre-flight — SelfSubjectAccessReview per watched resource (logs if watch denied)
Expand All @@ -16,10 +16,10 @@ sequenceDiagram
Router->>Kubernetes: start GC ticker goroutine (waits for cache sync, then runs on --gc-interval, default 5m)
Router->>Kubernetes: start controller-runtime manager (watch BGPRouter/BGPPeer/BGPAdvertisement/BGPVRFInstance/BGPPolicy/Secret/Node)
Note over Router: on each BGPRouter reconcile (not just the first)
Router->>GoBGP: lazy-start embedded server (only for --mode=tenant; listenPort defaults to 179, or -1 for outbound-only deployments)
Router->>GoBGP: lazy-start embedded server (listenPort defaults to 179, or -1 for outbound-only deployments)
Router->>GoBGP: StartBgp (ASN, RouterID from BGPRouter spec) — skipped if already started with the same values; Reconfigure (fresh BgpServer) if ASN/RouterID changed
Router->>GoBGP: apply peers, VRFs (route targets + kernel VRF wiring), start RIB monitor, apply EVPN advertisements, apply policies
Note over Router: on shutdown (SIGTERM/SIGINT): gRPC health server GracefulStop, manager exits when its signal context is cancelled. GoBGP has no explicit shutdown hook — it stops only because the process exits.
```

`--mode=fabric` uses an FRR runtime stub instead of GoBGP; `--mode=transit` is accepted by validation but returns an error at startup (not yet implemented). See [docs/router/configuration.md](router/configuration.md) for the full set of flags/environment variables.
See [docs/router/configuration.md](router/configuration.md) for the full set of flags/environment variables.
Loading