From 3fda5ce5d455355bb54823cff339fa228e75a590 Mon Sep 17 00:00:00 2001 From: YehoshuaSagron Date: Wed, 9 Sep 2026 12:58:47 +0300 Subject: [PATCH] Feat: Emit lineage.self.namespace; require namespace (contract v1.7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two workloads with the same name in two namespaces derived as ONE entity at the consumer: lineage.self.id is the last segment of the SPIFFE ID, and the data-governance sidecar algorithm keyed an entity on kind:self.id. A team1/weather-service and a team2/weather-service therefore shared one row, and every interaction of both pods pointed at it. That collision was made normative in #761 (round 5: contract v1.6.1 §4; round 6: the TestServiceLabel row "collides by design") as the documentation half — the reduction is deliberate and stays. This is the identity half: self.id alone was never the whole identity of a pod, so the missing fact is added rather than the shipped reduction changed. The plugin now emits lineage.self.namespace on both spans, from a new required `namespace` config key (or `namespace_file`, read once at start — for the file the kubelet projects from the pod's own metadata, the one source that is right in every copy of a ConfigMap shared across namespaces). The value is resolved before anything else in Init and must be an RFC 1123 DNS label: absent, blank, a "/" (which would make the consumer's {kind}:{namespace}/{self.id} key ambiguous), or any other shape refuses to start and leaves nothing behind — the treatment a blank self_id gets, because a name without a namespace is half an identity. It is never parsed out of the SPIFFE path (a registrar convention; a kit-attached pod has no SPIFFE ID). Neither identity fact is capped by max_attr_bytes any more — self.id was, and a truncated identity keys the pod on a name that is not its own; both are operator configuration, not caller input. lineage.self.id, its reduction, the §4 clause, the by-design test row and the span names are unchanged. The #761 round-6 question is closed alongside: a self_id made only of separators, which the reduction emits as-is, now refuses at start like a blank one, and a self_id_file carrying one keeps the plugin not-ready like a blank file. Wire contract v1.7.0 (vendored byte-identical with lab-data-governance): the attribute (§4), the keys (§6), and the consumer commitment that a pod's identity is the (namespace, self.id) pair, natural key {kind}:{namespace}/{self.id}, read from the request span (§7). Additive on the wire; breaking in configuration in both directions — a sidecar older than the key rejects a config that carries it, so image and ConfigMap flip together per pod (no published release carries the plugin yet, so no deployed configuration is affected). The attach kit writes its NAMESPACE, warns when a re-run patches nothing (no pod rolls; the old sidecar only hot-reloads), and the README covers the upgrade, the enrolled-workload route via namespace_file (a literal in the platform's shared ConfigMap would be wrong in every namespace but one), and three troubleshooting rows. The schema marks the key required for abctl and /v1/plugins; the Capabilities cite moves to v1.7. Tests: the namespace on both spans in both directions and through the real decode → Init → emit path (trimmed, otherwise verbatim); never capped; refusals for absent, blank, "/", uppercase, spaces, dots and 64 chars with 63 and padded accepted, and nothing left behind; refusal precedes the identity poll; namespace_file (projected file, inline wins; absent, blank — its own message — and non-label refuse); the schema's required flag; every Init fixture carries the key; the identity table gains the "/" and " // " rows and the file test a separators-only file. Tests that run a full Init shut the plugin down (provider and gRPC client), not only the provider. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01NJpNC6yQgZAdfiLAp3RksZ Signed-off-by: YehoshuaSagron --- authbridge/authlib/plugins/lineage/config.go | 24 ++ authbridge/authlib/plugins/lineage/plugin.go | 89 ++++++- .../authlib/plugins/lineage/plugin_test.go | 234 +++++++++++++++++- authbridge/docs/lineage-wire-contract.md | 47 +++- authbridge/docs/plugin-catalog.md | 6 +- authbridge/lineage-attach/README.md | 45 +++- authbridge/lineage-attach/RECIPE.md | 2 +- authbridge/lineage-attach/attach-lineage.sh | 8 +- authbridge/lineage-attach/sidecar-patch.sh | 20 +- 9 files changed, 439 insertions(+), 36 deletions(-) diff --git a/authbridge/authlib/plugins/lineage/config.go b/authbridge/authlib/plugins/lineage/config.go index 5b2e26778..62250518a 100644 --- a/authbridge/authlib/plugins/lineage/config.go +++ b/authbridge/authlib/plugins/lineage/config.go @@ -169,6 +169,30 @@ type Config struct { // not ready and skips every exchange, polling the file in background (see // Init), and /readyz names it meanwhile. Ignored when SelfID is set. SelfIDFile string `json:"self_id_file" description:"Read when self_id is empty; until it is readable the plugin is not ready and emits nothing. Refused at start only when self_id is also empty." default:"/shared/client-id.txt"` + + // Namespace is the Kubernetes namespace this workload runs in, emitted as + // the lineage.self.namespace fact on every span. Required: lineage.self.id + // alone is not an identity — the same workload name in two namespaces is + // two workloads, and the consumer keys entity identity on the + // (namespace, self.id) pair (wire contract §7). The attach kit writes its + // NAMESPACE here. It is never derived from the SPIFFE ID's path — that + // layout is a registrar convention, and the kit path has no SPIFFE ID at + // all. Empty, blank, or not an RFC 1123 DNS label (the only shape a + // namespace can have) refuses at start (see resolveNamespace); when empty, + // NamespaceFile is consulted instead. + Namespace string `json:"namespace" required:"true" description:"This workload's Kubernetes namespace (an RFC 1123 DNS label), emitted as lineage.self.namespace on every span; refused at start when empty or not a label. Alternatively namespace_file."` + + // NamespaceFile is a file carrying the namespace, read once at Init when + // Namespace is empty — meant for the path the kubelet projects from the + // pod's own metadata into every container that mounts the service-account + // volume, /var/run/secrets/kubernetes.io/serviceaccount/namespace. That + // is the one source that is correct in every copy of a ConfigMap shared + // across namespaces (the platform's per-namespace authbridge-runtime-config + // is rendered from one template and copied), where an inline literal would + // be confidently wrong in every namespace but one. No default and no + // poller: an absent file is a wrong path or a missing mount, and refuses + // at start like an empty value. Ignored when Namespace is set. + NamespaceFile string `json:"namespace_file" description:"Read when namespace is empty, once at start (e.g. /var/run/secrets/kubernetes.io/serviceaccount/namespace); absent, blank or not a DNS label refuses at start."` } func defaultConfig() Config { diff --git a/authbridge/authlib/plugins/lineage/plugin.go b/authbridge/authlib/plugins/lineage/plugin.go index dcde43d7e..08a356793 100644 --- a/authbridge/authlib/plugins/lineage/plugin.go +++ b/authbridge/authlib/plugins/lineage/plugin.go @@ -59,6 +59,7 @@ import ( "net" "os" "path" + "regexp" "strings" "sync/atomic" "time" @@ -205,7 +206,7 @@ func (p *LineageTelemetry) Capabilities() pipeline.PluginCapabilities { // The contract is cited major.minor only, deliberately: patch // revisions (v1.5.x) clarify prose and never change span semantics, // so a patch bump must not imply a producer change. - Description: "Emits two facts-only lineage spans per HTTP exchange (wire contract v1.6).", + Description: "Emits two facts-only lineage spans per HTTP exchange (wire contract v1.7).", } } @@ -237,7 +238,8 @@ func (p *LineageTelemetry) Init(ctx context.Context) error { // degrades honestly (abandoned / NULL / parent.source=wire or none). // Identity is the fact's subject — it has no degraded form, and a shared // placeholder would collapse every unidentified pod onto one entity row - // (entity id = uuid5("{kind}:{self.id}"), and entities is upsert-only). + // (entity id = uuid5("{kind}:{namespace}/{self.id}"), and entities is + // upsert-only). // // Failing closed is scoped to the span, not the process. An Init error // fails Pipeline.Start and the binary exits, every plugin in the chain @@ -254,13 +256,29 @@ func (p *LineageTelemetry) Init(ctx context.Context) error { // self_id, which no amount of waiting fixes; both return before the gRPC // client and batch-span-processor goroutine below exist, so a refused // start leaks nothing. + // + // The namespace — the other half of identity (config.go, Namespace) — + // is resolved FIRST. It is cheap, unconditional and needs no waiting + // (an inline value is known when the config is rendered; a file value + // sits where the kubelet projected it before this container started), + // so a refusal happens before the identity switch can log a poller it + // would never start, and before any goroutine or connection exists. + ns, err := resolveNamespace(p.cfg) + if err != nil { + return err + } + p.cfg.Namespace = ns + var pending string // self_id_file left for the poller to resolve switch { case p.cfg.SelfID != "": // Same reading as the file path: a blank value carries no identity - // and would key an entity on whitespace at the consumer. + // and would key an entity on whitespace at the consumer — and so + // does a value made only of separators ("/"), which serviceLabel + // returns as-is: a subject with no name is no subject (the #761 + // round-6 question, answered here). p.selfID = strings.TrimSpace(p.cfg.SelfID) - if p.selfID == "" { + if !hasIdentity(p.selfID) { return fmt.Errorf("lineage-telemetry: self_id %q carries no identity", p.cfg.SelfID) } case p.cfg.SelfIDFile != "": @@ -356,11 +374,11 @@ func (p *LineageTelemetry) Init(ctx context.Context) error { bgCtx, cancel := context.WithCancel(context.Background()) p.bgCancel.Store(&cancel) go p.awaitIdentity(bgCtx, pending, identityPollInterval) - slog.Info("lineage-telemetry: initialized, not ready until self_id_file resolves", "endpoint", endpoint, "self_id_file", pending) + slog.Info("lineage-telemetry: initialized, not ready until self_id_file resolves", "endpoint", endpoint, "self_id_file", pending, "namespace", p.cfg.Namespace) return nil } p.ready.Store(true) - slog.Info("lineage-telemetry: initialized", "endpoint", endpoint, "self_id", p.selfID) + slog.Info("lineage-telemetry: initialized", "endpoint", endpoint, "self_id", p.selfID, "namespace", p.cfg.Namespace) return nil } @@ -400,7 +418,7 @@ func (p *LineageTelemetry) awaitIdentity(ctx context.Context, path string, every p.ready.Store(false) return } - slog.Info("lineage-telemetry: identity loaded from self_id_file; recording spans", "path", path, "self_id", id) + slog.Info("lineage-telemetry: identity loaded from self_id_file; recording spans", "path", path, "self_id", id, "namespace", p.cfg.Namespace) return } if attempts++; logExportFailure(attempts) { @@ -419,12 +437,17 @@ func readIdentityFile(path string) (string, error) { if err != nil { return "", err } - if id := strings.TrimSpace(raw); id != "" { + if id := strings.TrimSpace(raw); hasIdentity(id) { return id, nil } return "", fmt.Errorf("file %s carries no identity", path) } +// hasIdentity is the one rule behind both identity sources: an identity is +// a string with at least one non-empty "/"-segment, so serviceLabel has a +// name to emit. Blank, and separator-only values such as "/", carry none. +func hasIdentity(id string) bool { return strings.Trim(id, "/") != "" } + // newTracerProvider builds the provider Init installs. AlwaysSample is // explicit and deliberate: lineage is an audit record, and under the SDK // default ParentBased sampler a caller sending a valid traceparent with the @@ -900,7 +923,14 @@ func spanKindFor(dir pipeline.Direction) trace.SpanKind { func (p *LineageTelemetry) baseAttrs(pctx *pipeline.Context, self, protocol string) []attribute.KeyValue { attrs := []attribute.KeyValue{ attribute.String("lineage.direction", pctx.Direction.String()), - p.capped("lineage.self.id", self), + // The two identity facts are not capped: an identity that reached the + // wire truncated would key the pod, at the consumer, on a name that is + // not its own. max_attr_bytes exists for caller-controlled values; + // both of these are operator configuration (self_id / self_id_file, + // namespace / namespace_file), and the namespace is bounded to a DNS + // label by Init besides. + attribute.String("lineage.self.id", self), + attribute.String("lineage.self.namespace", p.cfg.Namespace), attribute.String("lineage.protocol", protocol), } if pctx.Host != "" { @@ -1062,6 +1092,44 @@ func mcpTool(pctx *pipeline.Context) string { return "" } +// dnsLabel is the RFC 1123 label shape a Kubernetes namespace must have — +// the same check the attach kit applies to NAMESPACE before rendering it. +var dnsLabel = regexp.MustCompile(`^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$`) + +// resolveNamespace yields the namespace fact from the config: the inline +// key first, else namespace_file — read once, synchronously. There is no +// poller, unlike self_id_file: the file this knob exists for is the one the +// kubelet projects from the pod's own metadata before any container starts, +// so an absent file is a wrong path or a missing mount, not a race. +// +// The value is trimmed of surrounding whitespace and must be an RFC 1123 +// DNS label — the only shape a namespace can have. The consumer composes the +// entity key as "{kind}:{namespace}/{self.id}", so a "/" would make the key +// ambiguous, and a value longer than a label could reach the wire capped; +// both are refused here rather than emitted. Nothing is guessed: no source +// at all, an empty value, or a value of the wrong shape all refuse to start. +func resolveNamespace(cfg Config) (string, error) { + ns, source := strings.TrimSpace(cfg.Namespace), "namespace" + if ns == "" && cfg.NamespaceFile != "" { + // ReadCredentialFile already trims; absent or zero-length is its error. + raw, err := config.ReadCredentialFile(cfg.NamespaceFile) + if err != nil { + return "", fmt.Errorf("lineage-telemetry: namespace_file %q: %w", cfg.NamespaceFile, err) + } + if raw == "" { + return "", fmt.Errorf("lineage-telemetry: namespace_file %q carries no namespace", cfg.NamespaceFile) + } + ns, source = raw, "namespace_file "+cfg.NamespaceFile + } + if ns == "" { + return "", errors.New("lineage-telemetry: namespace is required (this workload's Kubernetes namespace): set namespace, or namespace_file to a file the kubelet projects") + } + if !dnsLabel.MatchString(ns) { + return "", fmt.Errorf("lineage-telemetry: %s: %q is not a DNS label (lowercase letters, digits and '-', 1-63 chars)", source, ns) + } + return ns, nil +} + // serviceLabel reduces a SPIFFE ID to its last non-empty path segment, or // returns selfID as-is if it is not a SPIFFE URI. Used for the lineage.self.id // fact and span names. The reduction is normative (contract §4): the consumer @@ -1071,7 +1139,8 @@ func mcpTool(pctx *pipeline.Context) string { // "spiffe://trust-domain/ns/team1/sa/weather-service" → "weather-service" // "weather-service" → "weather-service" // "spiffe://trust-domain/ns/team1/sa/agent/" → "agent" (trailing separator skipped) -// "/" → "/" (no non-empty segment: the input is returned unchanged) +// "/" → "/" (no non-empty segment: the input is returned unchanged; +// Init never admits such a value — see hasIdentity) // // selfID is never empty at the only call site: OnRequest runs only once ready, // and readiness is stored only after an identity resolved (Init or its diff --git a/authbridge/authlib/plugins/lineage/plugin_test.go b/authbridge/authlib/plugins/lineage/plugin_test.go index 29fc4d51b..45e8b1450 100644 --- a/authbridge/authlib/plugins/lineage/plugin_test.go +++ b/authbridge/authlib/plugins/lineage/plugin_test.go @@ -47,6 +47,7 @@ func newTestPlugin(t *testing.T) (*LineageTelemetry, *tracetest.InMemoryExporter p.tp = tp p.tracer = tp.Tracer("test") p.selfID = "weather-service" + p.cfg.Namespace = "team1" p.ready.Store(true) return p, exp } @@ -1227,6 +1228,11 @@ func TestConfigSchema_TracksConfig(t *testing.T) { if len(schema) != keys { t.Errorf("schema has %d fields, Config has %d json keys", len(schema), keys) } + // The operator-facing tooling (abctl templates, /v1/plugins) reads the + // required tag; the one key whose absence refuses boot must carry it. + if f, ok := byName["namespace"]; !ok || !f.Required { + t.Error("namespace is not marked required in the schema") + } } // The invocation action follows what happened to the message: the tracestate @@ -1422,7 +1428,7 @@ func TestConfigure_Defaults(t *testing.T) { func TestConfigure_DecodesKeptKeys(t *testing.T) { p := NewLineageTelemetry() - raw := json.RawMessage(`{"otel_endpoint":"http://collector:4317","capture_io":true,"self_id":"weather-service"}`) + raw := json.RawMessage(`{"otel_endpoint":"http://collector:4317","capture_io":true,"self_id":"weather-service","namespace":"team1"}`) if err := p.Configure(raw); err != nil { t.Fatalf("Configure: %v", err) } @@ -1435,6 +1441,9 @@ func TestConfigure_DecodesKeptKeys(t *testing.T) { if p.cfg.SelfID != "weather-service" { t.Errorf("self_id = %q", p.cfg.SelfID) } + if p.cfg.Namespace != "team1" { + t.Errorf("namespace = %q", p.cfg.Namespace) + } } // ---- helpers ---- @@ -1500,9 +1509,13 @@ func TestInit_RefusesToStartWithoutIdentity(t *testing.T) { cfg Config wantErr bool }{ - {"inline self_id starts", Config{OTelEndpoint: "localhost:4317", SelfID: "weather-service"}, false}, - {"blank inline self_id refuses", Config{OTelEndpoint: "localhost:4317", SelfID: " \n"}, true}, - {"no identity source refuses", Config{OTelEndpoint: "localhost:4317"}, true}, + {"inline self_id starts", Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfID: "weather-service"}, false}, + {"blank inline self_id refuses", Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfID: " \n"}, true}, + // Separators only: serviceLabel would emit "/" as an entity key — a + // subject with no name. Refused, like blank (#761 round 6). + {"slash-only inline self_id refuses", Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfID: "/"}, true}, + {"slashes-only inline self_id refuses", Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfID: " // "}, true}, + {"no identity source refuses", Config{OTelEndpoint: "localhost:4317", Namespace: "team1"}, true}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -1527,6 +1540,206 @@ func TestInit_RefusesToStartWithoutIdentity(t *testing.T) { } } +// TestInit_RefusesWithoutNamespace: the namespace is the other half of +// identity (config.go, Namespace) — the consumer keys an entity on +// (namespace, self.id), so a pod emitting only self.id would collapse onto +// the row of a same-named pod in another namespace. It is inline-only and +// known when the config is rendered, so there is no file to wait for: absent +// or blank refuses at boot, exactly as a blank self_id does — and so does a +// value that is not a DNS label: the consumer composes "{kind}:{ns}/{id}", so +// a "/" would make the key ambiguous, and a value longer than max_attr_bytes +// would reach the wire truncated. Surrounding whitespace alone is trimmed. +func TestInit_RefusesWithoutNamespace(t *testing.T) { + long := strings.Repeat("n", 64) + for _, tc := range []struct { + name string + ns string + wantErr bool + }{ + {"absent", "", true}, + {"blank", " \n\t", true}, + {"slash", "team1/team2", true}, + {"uppercase", "TEAM1", true}, + {"space inside", "team 1", true}, + {"dots", "team1.svc", true}, + {"leading hyphen", "-team1", true}, + {"64 chars", long, true}, + {"63 chars", long[:63], false}, + {"padded label", " team1\n", false}, + } { + t.Run(tc.name, func(t *testing.T) { + p := NewLineageTelemetry() + p.cfg = Config{OTelEndpoint: "localhost:4317", SelfID: "weather-service", Namespace: tc.ns} + err := p.Init(context.Background()) + if !tc.wantErr { + if err != nil { + t.Fatalf("Init refused a valid namespace %q: %v", tc.ns, err) + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + _ = p.Shutdown(ctx) // the provider AND the gRPC client Init stored + cancel() + if got := strings.TrimSpace(tc.ns); p.cfg.Namespace != got { + t.Errorf("stored namespace = %q, want %q", p.cfg.Namespace, got) + } + return + } + if err == nil { + t.Fatalf("Init succeeded with namespace %q", tc.ns) + } + if !strings.Contains(err.Error(), "namespace") { + t.Errorf("error does not name the key: %v", err) + } + if p.Ready() { + t.Error("plugin reports Ready after a refused Init") + } + // The refusal precedes everything else Init builds: no tracer + // provider, no poller, nothing to leak or to shut down. + if p.tp != nil || p.bgCancel.Load() != nil { + t.Error("a refused namespace left a tracer provider or a poller behind") + } + }) + } +} + +// TestInit_RefusesNamespaceBeforeIdentityPoll: the namespace check runs +// before the identity switch, so a config with an unreadable self_id_file +// AND no namespace is refused outright — the log never promises a poller +// that Init then fails to start, and no goroutine exists after the refusal. +func TestInit_RefusesNamespaceBeforeIdentityPoll(t *testing.T) { + p := NewLineageTelemetry() + p.cfg = Config{OTelEndpoint: "localhost:4317", SelfIDFile: t.TempDir() + "/absent"} + if err := p.Init(context.Background()); err == nil { + t.Fatal("Init succeeded with no namespace and an unreadable self_id_file") + } + if p.bgCancel.Load() != nil || p.tp != nil || p.Ready() { + t.Error("refused Init left a poller, a tracer provider, or readiness behind") + } +} + +// TestInit_NamespaceFile: the file source exists for the one ConfigMap that +// is shared across namespaces (the platform's), where an inline literal is +// wrong everywhere but one namespace and the kubelet-projected file is right +// everywhere. Read once, no poller; the inline key wins when set; the same +// shape rules apply; an absent file refuses rather than waits. +func TestInit_NamespaceFile(t *testing.T) { + dir := t.TempDir() + write := func(name, content string) string { + path := dir + "/" + name + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatal(err) + } + return path + } + for _, tc := range []struct { + name string + cfg Config + want string + wantErr string + }{ + {"projected file", Config{NamespaceFile: write("ns", "team2\n")}, "team2", ""}, + {"inline wins over file", Config{Namespace: "team1", NamespaceFile: write("ns2", "team2\n")}, "team1", ""}, + {"absent file refuses", Config{NamespaceFile: dir + "/missing"}, "", "/missing"}, + {"blank file refuses", Config{NamespaceFile: write("blank", " \n")}, "", "carries no namespace"}, + {"non-label file refuses", Config{NamespaceFile: write("bad", "Team-2\n")}, "", "not a DNS label"}, + } { + t.Run(tc.name, func(t *testing.T) { + p := NewLineageTelemetry() + tc.cfg.OTelEndpoint, tc.cfg.SelfID = "localhost:4317", "weather-service" + p.cfg = tc.cfg + err := p.Init(context.Background()) + if err == nil { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + _ = p.Shutdown(ctx) + cancel() + } + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("err = %v, want one naming %q", err, tc.wantErr) + } + return + } + if err != nil { + t.Fatalf("Init: %v", err) + } + if p.cfg.Namespace != tc.want { + t.Errorf("namespace = %q, want %q", p.cfg.Namespace, tc.want) + } + }) + } +} + +// TestNamespace_ThroughDecodeAndInit pins the real path — Configure (decode) +// then Init (trim) then an exchange — rather than a field the test wrote: +// a padded JSON value lands on both spans trimmed, and otherwise verbatim. +func TestNamespace_ThroughDecodeAndInit(t *testing.T) { + p := NewLineageTelemetry() + if err := p.Configure([]byte(`{"otel_endpoint":"localhost:4317","self_id":"weather-service","namespace":" team1\n"}`)); err != nil { + t.Fatalf("Configure: %v", err) + } + if err := p.Init(context.Background()); err != nil { + t.Fatalf("Init: %v", err) + } + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + _ = p.Shutdown(ctx) // the in-memory provider swapped in below, and the gRPC client + }) + // Swap the real exporter for an in-memory one so the spans can be read — + // after shutting the real provider down, so nothing Init built is orphaned. + { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + _ = p.tp.Shutdown(ctx) + cancel() + } + exp := tracetest.NewInMemoryExporter() + p.tp = sdktrace.NewTracerProvider(sdktrace.WithSyncer(exp)) + p.tracer = p.tp.Tracer("test") + run(t, p, fakeContext(pipeline.Outbound, http.Header{}), allow(200)) + req, resp := roleSplit(t, exp.GetSpans()) + for _, s := range []tracetest.SpanStub{req, resp} { + if got := attrStr(s, "lineage.self.namespace"); got != "team1" { + t.Errorf("%s: lineage.self.namespace = %q, want team1 (trimmed)", s.Name, got) + } + } +} + +// TestNamespace_NeverCapped: max_attr_bytes caps caller-controlled strings; +// an identity fact must not be truncated, or the consumer keys the pod on a +// name that is not its own. The namespace is bounded by Init's DNS-label +// check instead, so a tiny cap leaves it whole. +func TestNamespace_NeverCapped(t *testing.T) { + p, exp := newTestPlugin(t) + p.cfg.MaxAttrBytes = 4 + p.cfg.Namespace = "team-with-a-long-name" + run(t, p, fakeContext(pipeline.Inbound, http.Header{}), allow(200)) + req, _ := roleSplit(t, exp.GetSpans()) + if got := attrStr(req, "lineage.self.namespace"); got != "team-with-a-long-name" { + t.Errorf("lineage.self.namespace = %q, want the whole value", got) + } +} + +// TestNamespace_OnBothSpans: lineage.self.namespace rides beside +// lineage.self.id on the request AND the response span (contract §4 "both"), +// verbatim from config — never parsed out of a SPIFFE self_id — and the +// self.id fact is unchanged by it: the pair is composed by the consumer. +func TestNamespace_OnBothSpans(t *testing.T) { + for _, dir := range []pipeline.Direction{pipeline.Inbound, pipeline.Outbound} { + p, exp := newTestPlugin(t) + p.selfID = "spiffe://localtest.me/ns/team2/sa/weather-service" + p.cfg.Namespace = "team1" // deliberately NOT the SPIFFE path's ns: config is the fact + run(t, p, fakeContext(dir, http.Header{}), allow(200)) + req, resp := roleSplit(t, exp.GetSpans()) + for _, s := range []tracetest.SpanStub{req, resp} { + if got := attrStr(s, "lineage.self.namespace"); got != "team1" { + t.Errorf("%s %s: lineage.self.namespace = %q, want team1", dir, s.Name, got) + } + if got := attrStr(s, "lineage.self.id"); got != "weather-service" { + t.Errorf("%s %s: lineage.self.id = %q, want weather-service", dir, s.Name, got) + } + } + } +} + // TestInit_MissingSelfIDFileIsNotFatal: the default self_id_file is the // operator-mounted Secret, which can land after the pod starts. An Init error // fails Pipeline.Start and takes every other plugin in the chain down with it, @@ -1540,6 +1753,7 @@ func TestInit_MissingSelfIDFileIsNotFatal(t *testing.T) { }{ {"absent", nil}, {"present but blank", []byte(" \n\t\n")}, + {"present but separators only", []byte("/\n")}, } { t.Run(tc.name, func(t *testing.T) { path := t.TempDir() + "/client-id.txt" @@ -1590,7 +1804,7 @@ func TestShutdown_StopsIdentityPoll(t *testing.T) { for i := 0; i < 300; i++ { path := fmt.Sprintf("%s/client-id-%d.txt", dir, i) p := NewLineageTelemetry() - p.cfg = Config{OTelEndpoint: "localhost:4317", SelfIDFile: path} + p.cfg = Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfIDFile: path} if err := p.Init(context.Background()); err != nil { t.Fatalf("Init: %v", err) } @@ -1620,7 +1834,7 @@ func initPolling(t *testing.T, path string) *LineageTelemetry { identityPollInterval = 5 * time.Millisecond t.Cleanup(func() { identityPollInterval = restore }) p := NewLineageTelemetry() - p.cfg = Config{OTelEndpoint: "localhost:4317", SelfIDFile: path} + p.cfg = Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfIDFile: path} if err := p.Init(context.Background()); err != nil { t.Fatalf("Init with an unreadable self_id_file must not fail the process: %v", err) } @@ -1671,7 +1885,7 @@ func TestInit_ReadsSelfIDFile(t *testing.T) { t.Fatal(err) } p := NewLineageTelemetry() - p.cfg = Config{OTelEndpoint: "localhost:4317", SelfIDFile: path} + p.cfg = Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfIDFile: path} if err := p.Init(context.Background()); err != nil { t.Fatalf("Init: %v", err) } @@ -1855,7 +2069,7 @@ func TestConfig_BypassReplacesDefaults(t *testing.T) { // returns nil — since the closed socket itself is not introspectable here. func TestShutdown_ClosesConn(t *testing.T) { p := NewLineageTelemetry() - p.cfg = Config{OTelEndpoint: "localhost:4317", SelfID: "weather-service"} + p.cfg = Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfID: "weather-service"} if err := p.Init(context.Background()); err != nil { t.Fatalf("Init: %v", err) } @@ -1888,7 +2102,7 @@ func TestShutdown_NoInitIsSafe(t *testing.T) { // a torn-down tracer provider. func TestShutdown_ClearsReady(t *testing.T) { p := NewLineageTelemetry() - p.cfg = Config{OTelEndpoint: "localhost:4317", SelfID: "weather-service"} + p.cfg = Config{OTelEndpoint: "localhost:4317", Namespace: "team1", SelfID: "weather-service"} if err := p.Init(context.Background()); err != nil { t.Fatalf("Init: %v", err) } @@ -2111,7 +2325,7 @@ func TestInit_CAFile(t *testing.T) { for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { p := NewLineageTelemetry() - p.cfg = Config{OTelEndpoint: "collector.ns:4317", OTelCAFile: tc.caFile, SelfID: "x"} + p.cfg = Config{OTelEndpoint: "collector.ns:4317", Namespace: "team1", OTelCAFile: tc.caFile, SelfID: "x"} err := p.Init(context.Background()) if tc.wantErr { if err == nil { diff --git a/authbridge/docs/lineage-wire-contract.md b/authbridge/docs/lineage-wire-contract.md index db9ba59bc..e132dbdae 100644 --- a/authbridge/docs/lineage-wire-contract.md +++ b/authbridge/docs/lineage-wire-contract.md @@ -1,4 +1,4 @@ -# Lineage wire contract — two-span sidecar lineage (v1.6.3) +# Lineage wire contract — two-span sidecar lineage (v1.7.0) What the AuthBridge `lineage-telemetry` plugin emits, what it writes onto the wire, and what the data-governance `sidecar` interactions algorithm (ADR-0030) commits to when consuming it. @@ -174,7 +174,8 @@ handles `openinference.span.kind`. | `lineage.exchange.id` | both | `00f067aa0ba902b7` | the request span id, hex | | `lineage.role` | both | `request` \| `response` | which half this span is | | `lineage.direction` | both | `inbound` \| `outbound` | | -| `lineage.self.id` | both | `weather-service` | this workload's identity, from `self_id` or `self_id_file`, **reduced to its last non-empty `/`-segment**: a SPIFFE ID `spiffe://td/ns/team1/sa/agent` emits `agent`, and two identities that differ only above that segment emit the same value — the consumer keys entity identity on it (§7). The producer emits nothing without one: with no identity source configured it refuses to start, and while `self_id_file` is not yet readable it is not ready and skips every exchange (no span, no header) until the file resolves | +| `lineage.self.id` | both | `weather-service` | this workload's identity, from `self_id` or `self_id_file`, **reduced to its last non-empty `/`-segment**: a SPIFFE ID `spiffe://td/ns/team1/sa/agent` emits `agent`, and two identities that differ only above that segment emit the same value — the consumer keys entity identity on it (§7). The producer emits nothing without one: with no identity source configured it refuses to start, and while `self_id_file` is not yet readable it is not ready and skips every exchange (no span, no header) until the file resolves. Not capped by `max_attr_bytes` (an identity fact is never truncated; the value is operator configuration, not caller input) | +| `lineage.self.namespace` | both | `team1` | the Kubernetes namespace this workload runs in, from the `namespace` key or `namespace_file` (§6), trimmed of surrounding whitespace and otherwise verbatim; always an RFC 1123 DNS label (lowercase letters, digits and `-`, 1–63 chars), since the producer refuses any other shape. The other half of its identity: the consumer keys an entity on the (namespace, `self.id`) pair (§7), since the same `self.id` in two namespaces is two workloads. Never derived from the SPIFFE ID's path (a registrar convention, and the kit path has no SPIFFE ID); the producer refuses to start without one. Not capped by `max_attr_bytes`: an identity fact is never truncated | | `lineage.peer.host` | both, when present | `weather-tool-mcp.team1.svc:8000` | the Host/authority header. Outbound: the service being called. Inbound: the address this workload was reached on | | `lineage.protocol` | both | `a2a` \| `mcp` \| `inference` \| `http` | which parser matched, at fixed precedence `a2a` > `mcp` > `inference`; `http` = none. The precedence is load-bearing: the parsers are not mutually exclusive — `mcp-parser` attaches to any JSON-RPC body, including every a2a exchange — so an a2a hop is labeled `a2a`, never `mcp`. The payload reduction (§5) is keyed by this label, reading the same protocol's parser | | `lineage.parent.source` | request | `tracestate` \| `wire` \| `none` | which precedence in §3.2 chose the parent. An audit fact; the consumer derives nothing from it | @@ -239,12 +240,14 @@ construction. | `otel_ca_file` | — | PEM bundle to verify the collector's certificate against, for a private CA; implies `otel_tls`, and `otel_ca_file` with `otel_tls: false` is refused. An unreadable file, or one with no certificate, refuses to start | | `capture_io` | `false` | attach `input.value` / `output.value` | | `max_payload_bytes` | `4096` | producer-side cap on those two values; `0` or unset takes the default, `-1` attaches whole, any other negative is refused at start | -| `max_attr_bytes` | `256` | cap on every variable-content string attribute and the span name (§4); same `0` / `-1` / negative semantics as `max_payload_bytes` | +| `max_attr_bytes` | `256` | cap on every variable-content string attribute and the span name (§4), except the two identity facts `lineage.self.id` and `lineage.self.namespace`, which are operator configuration and never truncated; same `0` / `-1` / negative semantics as `max_payload_bytes` | | `mint_traceparent` | `true` | §3.3; `false` = a pure observer that never writes a `traceparent` | | `bypass_paths` | `/.well-known/*`, `/healthz`, `/readyz`, `/health` | path globs (`path.Match`; `*` does not cross `/`) that produce no spans, matched by the shared bypass package (query stripped, path normalized) — the same key and semantics as `jwt-validation` and `sparc` | | `bypass_hosts` | `otel-collector`, `otel-collector.*`, `jaeger`, `jaeger.*`, `zipkin`, `zipkin.*`, `prometheus`, `prometheus.*` | outbound host globs that produce no spans | -| `self_id` | — | this workload's identity (§4: reduced to its last `/`-segment); a blank value is refused at start | +| `self_id` | — | this workload's identity (§4: reduced to its last `/`-segment); a blank value, or one with no non-empty `/`-segment (`/`), is refused at start — no name, no subject | | `self_id_file` | `/shared/client-id.txt` | read when `self_id` is empty. Until it is readable and carries an identity the producer is not ready — every exchange skipped, nothing written to the wire — and it re-reads the file in the background, the sidecar's `/readyz` naming it meanwhile (a pod whose readiness probe uses `/readyz` stays out of rotation until the file lands — the `Readier` contract for a mounted credential, and in the stock chain `jwt-validation` already holds readiness on this same file); the process starts regardless, so the plugin never takes the sidecar's other plugins down over a late mount. Refused at start only when `self_id` is also empty | +| `namespace` | — | this workload's Kubernetes namespace, emitted as `lineage.self.namespace` (§4). Required (or `namespace_file`), and shape-checked: an absent or blank value, or one that is not an RFC 1123 DNS label, is refused at start — a `/` would make the consumer's `{kind}:{namespace}/{self.id}` key ambiguous. Resolved before anything else in the producer's start, so a refusal leaves nothing behind. The attach kit writes its `NAMESPACE`. A producer older than this key rejects a configuration that carries it (unknown keys are a boot error), so image and configuration change together | +| `namespace_file` | — | read once at start when `namespace` is empty; meant for `/var/run/secrets/kubernetes.io/serviceaccount/namespace`, which the kubelet projects from the pod's own metadata — the one source that is correct in every copy of a configuration shared across namespaces (the platform's per-namespace ConfigMap is rendered from one template and copied). Absent, blank or not a DNS label refuses at start; no default, no poller | Setting `bypass_paths` or `bypass_hosts` **replaces** the default list rather than extending it — the convention the `ibac`, `sparc` and `cpex` plugins use for their keys of the same name. An @@ -276,6 +279,26 @@ Unknown keys are a boot error. - Kinds and entity identity come from the facts only. `classify()` never requires `input.value` or `output.value`; bodyless exchanges produce complete, first-class interaction rows with NULL payload hashes, and the UI renders them like any other row. +- A pod's entity identity is the pair (`lineage.self.namespace`, `lineage.self.id`), read from the + request span (the response half is never consulted for identity): its natural key is + `{kind}:{namespace}/{self.id}` (so the row id, uuid5 of the natural key, splits with it), and the + namespace is stored again in `entities.namespace` for reading without parsing the key. The key + is parseable because a namespace is a DNS label and never contains `/`; a present value that is + not a DNS label (empty, padded, or any other shape) is a producer contract violation and halts + the derivation loudly, as a missing `self.id` does. The callee of an outbound hop takes both facts + from the callee's own echo span when it exists; the `peer.host` fallback, an LLM endpoint, a user + and an anonymous client have no namespace. A span with no `lineage.self.namespace` at all — one a + pre-v1.7 producer emitted, stored and replayable — keys its pod without a namespace, as v1.6 did: + absence is recorded, never filled in from `peer.host`, a SPIFFE path or the trace. Two + consequences follow and are accepted: traces recorded before the producer carried the fact keep + their un-namespaced identities for good (a re-derivation reproduces them), so a pod has one + entity row for its history and another from the cutover on unless the pre-v1.7 spans are dropped + before a replay; and during a mixed rollout a v1.6 callee's echo under a v1.7 caller keys that + callee without a namespace while its own v1.7 spans key it with one — one pod, two rows, for as + long as both trace sets exist. +- The natural key is a consumed vocabulary, not only a column: `lineage_metadata.data_sources` / + `.entities` store natural keys as text, the data-lineage traversal seeds on them, and any policy + matching an entity by name must match the namespaced form. - The consumer never welds a fragmented trace: an anchor whose parent is not a stored anchor derives as a root. - `content_kind` vocabulary stays ADR-0014-compatible; the classification processor consumes @@ -302,6 +325,22 @@ The producer must not emit these, and the consumer reads nothing from them. Version ladder, newest first. Each line is what changed on the wire or in the vocabulary; the mechanisms named as removed are not to be reintroduced. +- **v1.7.0** — `lineage.self.namespace` on both spans, from a new **required** `namespace` config + key or a `namespace_file` (absent, blank, or not a DNS label refuses at start, as a blank + `self_id` does; resolved first, so a refusal leaves nothing behind); a `self_id` made only of + separators (`/`), which the reduction would emit as-is, is now refused too, and a `self_id_file` + carrying one keeps the producer not-ready like a blank file. Neither identity fact is capped by + `max_attr_bytes` any more (`self.id` was): the pair the consumer keys on is never truncated. The reduction itself, its §4 clause + and the by-design collision of same-named workloads across namespaces on `self.id` alone are + unchanged — that clause is the documentation half (v1.6.1); the namespace is the identity half; the consumer keys a pod's + entity on (namespace, `self.id`) — natural key `{kind}:{namespace}/{self.id}` — and stores the + namespace in a new nullable `entities.namespace` column. Motivation: `self.id` is + the last segment of the SPIFFE ID, so `team1/weather-service` and `team2/weather-service` derived + as one entity, and every interaction of both pods pointed at one row. The namespace is a + configured fact, not parsed out of the SPIFFE path — the `ns/…/sa/…` layout is a registrar + convention, and a kit-attached pod has no SPIFFE ID at all. `lineage.self.id`, its reduction, and + span names are unchanged. Additive on the wire, breaking in configuration: every existing + `lineage-telemetry` block needs the key. - **v1.6.3** — an unreadable or blank `self_id_file` no longer refuses to start: the producer starts not-ready, skips every exchange (no span, no header) and re-reads the file until an identity appears. Until now the refusal failed the whole sidecar — every plugin in its chain — over a diff --git a/authbridge/docs/plugin-catalog.md b/authbridge/docs/plugin-catalog.md index ed5c2dc12..a404d4932 100644 --- a/authbridge/docs/plugin-catalog.md +++ b/authbridge/docs/plugin-catalog.md @@ -148,12 +148,14 @@ denial by a plugin ordered before it emits no spans. - `otel_ca_file` (string) — PEM bundle to verify the collector's certificate against (a private CA, e.g. cert-manager issued). Implies `otel_tls`; with an explicit `otel_tls: false` it is refused; an unreadable file or one with no certificate refuses to start. Default: system roots. - `capture_io` (bool) — attach the parsed request/response content as `input.value` / `output.value`. Default `false`. - `max_payload_bytes` (int) — cap on those two values, cut on a UTF-8 boundary with a `…[truncated]` marker; `0` or unset takes the default, `-1` attaches whole, any other negative is refused at start. Default `4096`. -- `max_attr_bytes` (int) — cap on every variable-content string attribute (`url.path`, `lineage.peer.host`, `mcp.tool`, …) and the span name; same `0` / `-1` / negative semantics as `max_payload_bytes`. Default `256`. +- `max_attr_bytes` (int) — cap on every variable-content string attribute (`url.path`, `lineage.peer.host`, `mcp.tool`, …) and the span name — except the two identity facts `lineage.self.id` and `lineage.self.namespace`, which are operator configuration and never truncated; same `0` / `-1` / negative semantics as `max_payload_bytes`. Default `256`. - `mint_traceparent` (bool) — forward a `traceparent` naming this request span when the request carried no valid one; `false` = a pure observer that writes no `traceparent`. Default `true`. - `bypass_paths` (`[]string`) — path globs (`path.Match`, query stripped, path normalized — the shared bypass matcher `jwt-validation` and `sparc` use) that produce no spans. Default `/.well-known/*`, `/healthz`, `/readyz`, `/health`. Setting either bypass key replaces its default list rather than extending it, as in `ibac` / `sparc` / `cpex`; an entry matching everything is refused at start. - `bypass_hosts` (`[]string`) — outbound host globs (`path.Match`, port stripped, case folded) that produce no spans; ignored inbound, where `Host` is caller-controlled. Default `otel-collector`, `otel-collector.*`, `jaeger`, `jaeger.*`, `zipkin`, `zipkin.*`, `prometheus`, `prometheus.*`. -- `self_id` (string) — this workload's identity, emitted as `lineage.self.id`; a blank value is refused at start. +- `self_id` (string) — this workload's identity, emitted as `lineage.self.id` (a SPIFFE ID reduced to its last path segment); a blank value, or one with no non-empty `/`-segment (`/`), is refused at start. - `self_id_file` (string) — read when `self_id` is empty. Until it is readable and carries an identity the plugin is not ready and skips every exchange (no span, no header), re-reading the file in the background while `/readyz` names it — the same handling `jwt-validation` gives this path, so a late Secret mount never fails the sidecar (a pod probing `/readyz` stays out of rotation until the file lands). Refused at start only when `self_id` is also empty. Default `/shared/client-id.txt`. +- `namespace` (string) — this workload's Kubernetes namespace (an RFC 1123 DNS label), emitted as `lineage.self.namespace` on every span: the other half of its identity, since `self_id` is the last segment of a SPIFFE ID and the same segment in two namespaces is two workloads. **Required** (or `namespace_file`) — absent, blank, or not a DNS label is refused at start; never derived from the SPIFFE path. The attach kit writes its `NAMESPACE`; a sidecar older than this key rejects a config that carries it, so image and config flip together. +- `namespace_file` (string) — read once at start when `namespace` is empty; meant for `/var/run/secrets/kubernetes.io/serviceaccount/namespace`, the file the kubelet projects from the pod's own metadata — the one source that is right in every copy of a ConfigMap shared across namespaces (the platform's `authbridge-runtime-config`), where an inline literal would be wrong everywhere but one. Absent, blank, or not a DNS label refuses at start; no default, no poller. ## `litellm-budget-track` diff --git a/authbridge/lineage-attach/README.md b/authbridge/lineage-attach/README.md index 329889181..8bfe8d092 100644 --- a/authbridge/lineage-attach/README.md +++ b/authbridge/lineage-attach/README.md @@ -167,10 +167,38 @@ When the platform injects its own AuthBridge sidecar (an `AgentRuntime` CR), lineage is enabled for the whole namespace by adding the three parsers + `lineage-telemetry` to both directions of the operator-rendered `authbridge-runtime-config` ConfigMap. Leave `self_id` unset — each pod -resolves its identity from the operator-mounted credential. The propagation -half is unchanged, and a platform upgrade re-renders the ConfigMap; re-apply -after one. This route is described here, not exercised: nothing in this kit -generates that edit. +resolves its identity from the operator-mounted credential. For the namespace +fact (required; it is what keeps two same-named pods in two namespaces two +entities at the consumer, wire contract §7) do **not** write a literal +`namespace: team1` into that ConfigMap: the platform chart renders it from one +template for every agent namespace and the operator copies the release +namespace's ConfigMap into namespaces that lack one, so a literal would be +confidently wrong in every namespace but one. Use the file source instead, +which is correct in every copy: + +```yaml +- name: lineage-telemetry + config: + otel_endpoint: "otel-collector.rossoctl-system.svc.cluster.local:4317" + namespace_file: /var/run/secrets/kubernetes.io/serviceaccount/namespace +``` + +That file is projected by the kubelet from the pod's own metadata into every +container that mounts the service-account volume; if the injected sidecar +does not mount it, the plugin refuses to start (loudly, naming the path) +rather than guess. The propagation half is unchanged, and a platform upgrade +re-renders the ConfigMap; re-apply after one. This route is described here, +not exercised: nothing in this kit generates that edit. + +**Upgrading across the namespace key.** A sidecar built before the key +rejects any config that carries it (unknown keys are a boot error), and one +built with it refuses any config that lacks it — so the image and the +ConfigMap flip together, per pod, never one before the other. On the kit +route that is one command: re-run `sidecar-patch.sh` with a `SIDECAR_IMAGE` +that carries the key; it rewrites the ConfigMap and patches the Deployment, +and the image change rolls the pod. A re-run with an unchanged image rolls no +pod, and the running sidecar only hot-reloads the ConfigMap — the script says +so and names the log line to check. --- @@ -242,10 +270,14 @@ The generated ConfigMap's plugin entry: otel_endpoint: "otel-collector.rossoctl-system.svc.cluster.local:4317" # host:port; https:// prefix turns on TLS capture_io: false # the plugin's default; CAPTURE_IO=true attaches the parsed content — PII lives in it self_id: "" # ALWAYS set (SELF_ID, default: the Deployment name) — see below + namespace: "" # ALWAYS set (NAMESPACE): the plugin refuses to start without it # max_payload_bytes: 4096 — the plugin's default cap on a captured value (MAX_PAYLOAD_BYTES) ``` -`self_id` is always emitted: the plugin's `self_id_file` fallback (the +`namespace` is the pod's Kubernetes namespace and rides on every span as +`lineage.self.namespace`; the consumer keys an entity on the (namespace, +`self_id`) pair, so the same Deployment name in two namespaces stays two +entities. `self_id` is always emitted: the plugin's `self_id_file` fallback (the operator-mounted credential, which can race its own Secret and fail the sidecar's boot) is deliberately never used on a ConfigMap this kit generates. The plugin's `bypass_paths` / `bypass_hosts` keep infrastructure noise out @@ -298,6 +330,9 @@ BAKE — once per app image ATTACH — once per Deployment |---|---| | No spans at all | Wrong `OTEL_ENDPOINT`, or the sidecar image predates the plugin — read the `envoy-proxy` container's log. | | `envoy-proxy` restarts with `unknown plugin "lineage-telemetry"` | The published image, until a release carries the plugin. Build from this repo (RECIPE step 1); the printed back-out line meanwhile. The patch pulls `IfNotPresent`, so a node that cached an older `:latest` keeps it. | +| `envoy-proxy` restarts with `json: unknown field "namespace"` | A ConfigMap from this kit against a sidecar image built before the namespace key. Re-run `sidecar-patch.sh` with a `SIDECAR_IMAGE` that carries it ("Upgrading across the namespace key"). | +| `envoy-proxy` restarts with `namespace is required` or `is not a DNS label` | A sidecar that carries the key against a ConfigMap that lacks it or hand-carries a value that is not a namespace. Re-run `sidecar-patch.sh` (it renders `NAMESPACE`, already validated as a DNS label). | +| Spans arrive without `lineage.self.namespace` after an attach that printed "attached" | The Deployment patch was a no-op (same image, same knobs), so no pod rolled and the old sidecar refused the hot-reload. The script prints a NOTE with the log line to check; re-run with a matching `SIDECAR_IMAGE`. | | Only inbound hops, never outbound | `proxy-init` did not install its iptables rules — its log. | | Outbound hops fragment (`lineage.parent.source=none` on the pod's outbound hops) | `traceparent` not propagating: the app container lacks `LINEAGE_PROPAGATE=1` (the patch sets it with `APP_CONTAINER`; an operator-owned Deployment needs `SELF_ACTIVATE=1`), or the call runs in a worker thread (the `threading` instrumentor is bundled), or the client library is outside the envelope. Only the entry hop dangling is expected. | | The app cannot reach its database / mail server after the patch | A plaintext non-HTTP port went through the outbound HTTP codec — `OUTBOUND_PORTS_EXCLUDE` it. | diff --git a/authbridge/lineage-attach/RECIPE.md b/authbridge/lineage-attach/RECIPE.md index 46e9262e5..bf81ed7c9 100644 --- a/authbridge/lineage-attach/RECIPE.md +++ b/authbridge/lineage-attach/RECIPE.md @@ -87,7 +87,7 @@ run the back-out line printed above, then read the sidecar log (step 4). ```sh kubectl -n $NS logs deploy/$DEPLOY -c envoy-proxy | grep 'lineage-telemetry: initialized' ``` -Pass: `… endpoint= self_id=`. +Pass: `… endpoint= self_id= namespace=`. Then one request **from inside the cluster** (a port-forward bypasses the sidecar) with a trace id you choose: ```sh diff --git a/authbridge/lineage-attach/attach-lineage.sh b/authbridge/lineage-attach/attach-lineage.sh index c4306fdc9..a6e39e545 100755 --- a/authbridge/lineage-attach/attach-lineage.sh +++ b/authbridge/lineage-attach/attach-lineage.sh @@ -50,7 +50,10 @@ # Variables: # NAME (required) the target Deployment; also names the ConfigMap # (authbridge-lineage-config-NAME) and defaults SELF_ID -# NAMESPACE default team1 +# NAMESPACE default team1; also emitted as the plugin's namespace key +# (lineage.self.namespace on every span — the other half of +# the entity identity, required by the plugin since cortex +# contract v1.7) # SELF_ID lineage identity on every span (default NAME) # APP_CONTAINER the app container to set LINEAGE_PROPAGATE=1 on. MUST name # an existing container: a strategic merge ADDS a stub for an @@ -249,7 +252,8 @@ build_plugin_entry() { config: otel_endpoint: "'"${OTEL_ENDPOINT}"'" capture_io: '"${CAPTURE_IO}"' - self_id: "'"${SELF_ID}"'"' + self_id: "'"${SELF_ID}"'" + namespace: "'"${NAMESPACE}"'"' # The plugin's own default applies when unset; an explicit value is emitted. [ -z "$MAX_PAYLOAD_BYTES" ] || lineage_plugin="${lineage_plugin} max_payload_bytes: ${MAX_PAYLOAD_BYTES}" diff --git a/authbridge/lineage-attach/sidecar-patch.sh b/authbridge/lineage-attach/sidecar-patch.sh index 5022bca4d..730875166 100755 --- a/authbridge/lineage-attach/sidecar-patch.sh +++ b/authbridge/lineage-attach/sidecar-patch.sh @@ -40,7 +40,8 @@ # # Env — read here: # DEPLOY target Deployment (required) -# NAMESPACE default team1 +# NAMESPACE default team1; also written into the plugin config as its +# identity namespace (lineage.self.namespace on every span) # SELF_ID lineage identity (default: DEPLOY) # APP_CONTAINER the app container to switch propagation on (optional) # Env — inherited by attach-lineage.sh and validated there (see its header): @@ -213,13 +214,28 @@ apply() { cm_existed=1 fi kubectl apply -f - <<<"$cm" - kubectl patch deploy "$DEPLOY" -n "$NAMESPACE" --type strategic --patch "$patch" || { + patched="$(kubectl patch deploy "$DEPLOY" -n "$NAMESPACE" --type strategic --patch "$patch")" || { # Only a failure the dry-run could not predict lands here (e.g. a 409 # from a concurrent write). Nothing else was written this run except, # possibly, the ConfigMap — remove it only if this run created it. [ "$cm_existed" = "1" ] || kubectl delete cm -n "$NAMESPACE" "authbridge-lineage-config-$DEPLOY" exit 1 } + echo "$patched" + # A re-run whose patch changes nothing (same images, same knobs) rolls no + # pod: the running sidecar picks the rewritten ConfigMap up by hot-reload, + # and a reload the binary rejects — a pre-v1.7 sidecar refusing the + # `namespace` key, say — is logged and otherwise silent, the old pipeline + # kept. "attached" below would then be true of the objects and false of + # the spans. Say so, and name the check. + case "$patched" in + *"(no change)"*) + echo "NOTE: the Deployment was already patched; no pod rolled. The ConfigMap change reaches" >&2 + echo " the running sidecar by hot-reload only. Confirm with:" >&2 + echo " kubectl -n $NAMESPACE logs deploy/$DEPLOY -c envoy-proxy | grep 'lineage-telemetry: initialized'" >&2 + echo " (expect namespace=$NAMESPACE); a sidecar older than the plugin's config refuses the reload" >&2 + echo " and keeps its previous pipeline — re-run with a matching SIDECAR_IMAGE, which rolls the pod." >&2 ;; + esac # Said before the wait: a rollout that never completes still needs this line. # CM after the patch: pods of a revision that still mounts it cannot start. echo ">> back out: kubectl -n $NAMESPACE patch deploy/$DEPLOY --type strategic -p '$undo' && kubectl -n $NAMESPACE delete cm authbridge-lineage-config-$DEPLOY"