Skip to content

[FEATURE] ClickHouse: add trace query plugin - #813

Open
erdincka wants to merge 3 commits into
perses:mainfrom
erdincka:feat/clickhouse-trace-query
Open

erdincka wants to merge 3 commits into
perses:mainfrom
erdincka:feat/clickhouse-trace-query

Conversation

@erdincka

@erdincka erdincka commented Sep 12, 2026

Copy link
Copy Markdown

Description

Adds ClickHouseTraceQuery, a TraceQuery plugin in the ClickHouse plugin module, so traces stored in ClickHouse can be displayed with the existing Trace Table and Tracing Gantt Chart panels. Until now the module had log and time series queries only.

Relates to perses/perses#4202 (ClickHouse traces as a datasource): this is the query plugin a ClickHouse trace explorer would build on.

How it works

The plugin reads the schema created by the OpenTelemetry Collector ClickHouse exporter, and follows the TraceData contract used by the Tempo plugin: a trace ID returns a trace, anything else returns search results.

  • Trace ID (16 or 32 hex characters, after variable substitution): the plugin builds the SQL, reads every span of the trace from table (default otel_traces, optionally database.table), and converts the rows to OTLP, grouped by resource and instrumentation scope. Drill-down links such as ?var-traceId=${traceId} work as they do with Tempo.
  • SQL: returns one row per span, with the exporter's column names (TraceId, Timestamp, Duration, ParentSpanId, SpanName, ServiceName, StatusCode). The plugin uses it as a subquery, and ClickHouse groups the spans into traces: one row per trace with the root span, the start and end time, and the span and error counts per service, newest first, with LIMIT limit + 1. The extra trace is how the panels know more traces match, as in the Tempo and Jaeger plugins. {start} and {end} are replaced as in the log and time series queries.
# Trace Table: traces that went through the payment service
kind: ClickHouseTraceQuery
spec:
  query: |
    SELECT TraceId, ParentSpanId, SpanName, ServiceName, Timestamp, Duration, StatusCode
    FROM otel.otel_traces
    WHERE Timestamp BETWEEN '{start}' AND '{end}'
      AND TraceId IN (
        SELECT TraceId FROM otel.otel_traces
        WHERE ServiceName = 'payment' AND Timestamp BETWEEN '{start}' AND '{end}'
      )
---
# Tracing Gantt Chart: the trace selected in the traceId variable
kind: ClickHouseTraceQuery
spec:
  query: $traceId
  table: otel.otel_traces

Implementation notes

  • Timestamps. Both queries convert timestamps to nanosecond strings in SQL (toUnixTimestamp64Nano). Otherwise DateTime64 values are rendered in the server's timezone, and nanoseconds since the epoch exceed Number.MAX_SAFE_INTEGER, so they are summed and subtracted with BigInt.
  • SQL built by the plugin. The trace ID is validated and normalized (lowercase, padded to 32 characters, as the exporter stores it), and table must be a plain identifier (checked in CUE and at runtime) before either goes into SQL. User-written search queries keep the existing behavior of the ClickHouse queries.
  • Limit. limit is applied by ClickHouse, not after fetching: only the traces that are displayed cross the wire. On the verification data, the search panel returns 8 rows instead of 48 span rows, and a search over 48 traces with limit: 20 returns 21 rows instead of 168. The cost is that the search query is used as a subquery: all the documented columns are required, it has to be a single SELECT, and a query ending with a FORMAT clause is rejected with an explicit error. A trailing ; is removed.
  • Output format. Both generated queries end with an explicit FORMAT JSON. The shared ClickHouse client only appends it when the query doesn't contain the word FORMAT, which a search query may well do (formatDateTime(...)), and would otherwise get TabSeparated back.
  • Errors. The ClickHouse client returns status: 'error' and logs the server's message to the console, so the plugin throws rather than showing an empty panel. An unknown trace ID also throws, as in the Jaeger plugin.
  • Exporter details. SpanKind and StatusCode are accepted as the exporter writes them (Server, Error) and as OTLP enum names (SPAN_KIND_SERVER, STATUS_CODE_ERROR). Attributes from the default Map schema are strings; with the exporter's json: true schema the value types are kept and mapped to the matching OTLP value (intValue, doubleValue, boolValue, arrayValue), and the nesting that ClickHouse's JSON type makes of dotted attribute names (http.response.status_code comes back as {http: {response: {status_code: 200}}}) is flattened back into the original names. 64-bit integers are accepted as JSON numbers or strings, since ClickHouse quotes them or not depending on output_format_json_quote_64bit_integers.

Changes

Path Change
clickhouse/src/queries/click-house-trace-query/ Plugin, options editor, data fetching and conversion, types, tests
clickhouse/schemas/queries/click-house-trace-query/ CUE schema, with valid and invalid test cases
clickhouse/sdk/go/query/trace/ Go SDK builder (Query, Datasource, Table, Limit) and tests
clickhouse/package.json, rsbuild.config.ts, src/queries/index.ts Register and expose the plugin
docs/clickhouse/ Overview, data model (including the trace ID lookup and the search query columns), Go SDK page

Tests

  • 26 vitest tests. Trace lookup: SQL and its output format, OTLP conversion (resource and scope grouping, both span kind and status spellings, nanosecond end times, events, links, typed and flattened attributes from the JSON schema), trace ID normalization, variables, table validation, trace not found. Search: the generated query and its limit, the summary rows it returns, time range placeholders, more results than the limit, a 16 character SQL query running as a search, formatDateTime in the query, a trailing semicolon, a rejected FORMAT clause, unknown root service, counts of an empty service name and a service named unknown, ClickHouse errors. Editor: table committed on blur, limit.
  • 2 Go tests for the SDK builder.
  • 5 schema test cases: 2 valid, and 3 invalid (empty query, table that is not an identifier, non-positive limit).

Verification

CI-equivalent run. I ran the steps of react.yml, cue.yml, go.yml, doc.yml and ci.yml at this branch's commit, in a container with the versions CI pins (Go 1.27.1, Node 24, CUE v0.16.1, percli v0.54.0, golangci-lint v2.13.2, mdox). All 17 steps passed:

Steps Result
npm ci, npm run lint, npm run format:check, npm run type-check, npm run test Pass (clickhouse: 6 test files, 37 tests)
make checkformat-cue, make lint-plugins, make test-schemas-plugins, make tidy-modules with no cue.mod diff Pass (the 2 valid and 3 invalid click-house-trace-query cases are evaluated)
make test, go test ./... in clickhouse, make golangci-lint, make checklicense Pass
make checkdocs Pass
percli plugin build for clickhouse, tracetable and tracingganttchart Pass

The advisory React Doctor scan (react-doctor.yml) reports no findings in the new files.

End to end. OTLP/HTTP → OpenTelemetry Collector contrib 0.160.0 (ClickHouse exporter, create_schema: true) → ClickHouse 25.8 → Perses (main image of 2026-09-10), with a ClickHouseDatasource going through the Perses proxy and the plugin archives built from this branch. The data: 8 traces across three services (with errors, exception events and span links), plus 40 traces from telemetrygen.

  • The ClickHouse plugin module loads with TraceQuery:ClickHouseTraceQuery, and a dashboard using it is provisioned.
  • POST /api/validate/dashboards accepts valid specs, and rejects a table that is not an identifier, limit: 0, an empty query and an unknown field, each with the corresponding CUE error.
  • The trace lookup SQL, run against the table created by the exporter, returns the types the conversion expects: maps as JSON objects, events and links as arrays, nanoseconds as strings.
  • A Trace Table with a search query lists the traces with their span and error counts. With 48 traces and limit: 20, it shows the "Not all matching traces are currently displayed" notice. ClickHouse's query_log confirms the limit is applied there: 21 rows returned for that panel, and 8 for the search on the payment service.
  • A Tracing Gantt Chart with query: $traceId renders the trace: span hierarchy, error statuses, attributes and events. Following a Trace Table link (?var-traceId=${traceId}) shows the selected trace.
  • An unknown trace ID shows "Trace ffffffffffffffffffffffffffffffff was not found in otel.otel_traces." in the panel. It is the only error in the browser console.

Review round two (Copilot's comments), checked on the same stack:

  • A Trace Table whose search query contains formatDateTime(...) renders its traces. Sent through the Perses proxy, the same query without the explicit FORMAT JSON comes back as text/tab-separated-values, which is what broke before.
  • A table created verbatim from the exporter's traces_json_table.sql (json: true schema, SpanAttributes JSON), holding a trace with a number, a float, a boolean, an array and nested keys, renders in the Tracing Gantt Chart with the attributes typed and under their original dotted names: http.response.status_code 200, http.request.body.size 12.5, retry true, tags checkout, web, nested.feature.flag on. ClickHouse returns those attributes as nested objects, which the plugin flattens back.

I can share the Compose setup used for this (collector config, seed script, provisioning) if it helps the review.

Design decision to review: trace ID lookups are not bounded by time

Important

A trace ID lookup reads the spans with WHERE TraceId = '…' from table, with no time bound, so that a trace opens from a link whatever the time range of the dashboard. It relies on the bloom filter index the exporter creates on TraceId.

The alternative is to first read the start and end time of the trace from the exporter's <table>_trace_id_ts table (otel_traces_trace_id_ts by default), and bound the scan with them, which lets ClickHouse skip the partitions outside the trace on large tables. I did not do it because that table only exists when the exporter created the schema, so the lookup would fail on custom tables and views that otherwise have the right columns.

This is the part of the design most likely to need a change, for example an option to use that table, or using it by default. The choice is documented in the data model docs ("Trace ID lookup") and in a comment on the query builder, so it is easy to revisit. Happy to change it if you prefer the other way.

Noticed while working on this, not changed in this PR

  • The existing ClickHouse Go SDK docs don't match the SDK. docs/clickhouse/go-sdk/datasource.md, log-query.md and timeseries-query.md import github.com/perses/perses-plugins/clickhouse/sdk/go/v1/..., while the SDK packages are github.com/perses/plugins/clickhouse/sdk/go/datasource, .../query/log and .../query/time-series. The two query pages also use builders that don't exist (query.LogQuery, query.TimeSeriesQuery, query.Format): the SDK has log.ClickHouseLogQuery and timeseries.ClickHouseTimeSeriesQuery, with Query and Datasource options only. Likewise, docs/clickhouse/model.md documents an optional format field for ClickHouseTimeSeriesQuery and ClickHouseLogQuery, which their CUE schemas (closed, with datasource and query only) reject. I didn't find an existing issue about it. I left these pages unchanged to keep this PR focused (the new trace-query.md uses the actual packages), and I'm happy to fix them in a follow-up.
  • The Tracing Gantt Chart keeps the viewport and the selected span of the previous trace when its trace changes in place, for example after following a Trace Table link to the same dashboard: TracingGanttChart initializes both with useState, and the panel doesn't key the component by trace. The lower timeline then shows offsets from the previous trace, and the details pane the previously selected span. It happens with any trace query plugin, so it isn't addressed here. The drill-down screenshot below was taken after reloading the page.

Screenshots

Perses main with the plugin archives built from this branch, reading traces written by the OpenTelemetry Collector ClickHouse exporter.

Dashboard: two Trace Tables with search queries, and a Tracing Gantt Chart with query: $traceId

01-dashboard

Span details converted from the exporter's columns: kind, status, attributes, resource, scope, events

03-span-detail

Query editor: datasource, trace ID or SQL, trace table, max traces

05-query-editor

Search limit: limit: 20 with 48 matching traces

02-limit-notice

A Trace Table link followed to the Gantt chart (after a page reload, see the Tracing Gantt Chart note above)

04-drilldown

Unknown trace ID

06-trace-not-found

Review round two: a search using formatDateTime, and a trace from the exporter's JSON schema

07-review-checks

JSON schema attributes in the detail pane, typed and flattened

08-json-schema-attributes

Checklist

  • Pull request has a descriptive title and context useful to a reviewer.
  • Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
    following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
  • All commits have DCO signoffs.

UI Changes

  • Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
  • Code follows the UI guidelines.

🤖 Generated with Claude Code

Add ClickHouseTraceQuery, a TraceQuery plugin reading traces stored with
the OpenTelemetry Collector ClickHouse exporter schema, so the Trace
Table and Tracing Gantt Chart panels can be used with ClickHouse.

Like the Tempo trace query, a trace ID returns the whole trace as OTLP
and any other query returns search results. A search query is SQL
returning one row per span, grouped by TraceId into search results.

Trace ID lookups are deliberately not bounded by time, so that a trace
opens from a link whatever the dashboard time range. The data model docs
explain the trade-off with the exporter's trace ID lookup table.

Includes the CUE schema with valid and invalid test cases, the Go SDK
builder, unit tests and docs.

Refs perses/perses#4202

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Erdinc Kaya <erdincka@msn.com>
@erdincka
erdincka marked this pull request as ready for review September 12, 2026 13:05
@erdincka
erdincka requested review from a team, AntoineThebaud and Nexucis as code owners September 12, 2026 13:05
@erdincka
erdincka requested review from jgbernalp and removed request for a team September 12, 2026 13:05
type: 'info',
message: 'Not all matching traces are currently displayed. Increase the result limit to view additional traces.',
});
searchResult.splice(limit);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although the trade-off is documented, can a query limit be added and configured by a field in the query spec?, without it we just fetch all the results to remove them after fetching.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, thanks. limit was already in the spec (the "Max traces" field) but was only applied after fetching. I have tested 2 resolutions:

  1. The search now runs the query as a subquery and does the grouping in ClickHouse: one row per trace (root span, start and end time, span and error counts per service), newest first, LIMIT limit + 1 to detect hasMoreResults as the Tempo plugin does. On my test data, a search over 48 traces now returns 21 rows instead of 168 span rows. The trade-off is that all seven columns are now required and the query has to be a single SELECT without its own FORMAT. Both are documented, and a FORMAT clause is rejected with an explicit error.
  2. A {limit} placeholder the user writes into their own SQL, like {start}/{end}. It's a much smaller change, but the pushdown only works if the user puts it in the right place, which is the trace ID subquery (at span level it would cut traces short), and the browser would still fetch every span of those traces.
    Would you agree to go with Option 1?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not an expert in ClickHouse, so I lean on your expertise. But option 1 seems more reasonable as the user might not know which one is the limit they need to use.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pushed a separate commit for this (including the test and doc update for the trace id thread). The search query is now used as a subquery and ClickHouse does the grouping and the limit (buildSearchQuery in get-click-house-trace-data.ts): one row per trace with the root span, start and end time, and span and error counts per service, newest first, with LIMIT limit + 1.

Review feedback: the search fetched every matching span and grouped them
in the browser, so the limit only dropped traces after the fact.

The search query is now used as a subquery. ClickHouse groups the spans
per trace, orders them newest first and applies LIMIT limit + 1, the
extra row telling the panels that more traces match. Only the traces
that are displayed cross the wire, and start and end times come back as
nanoseconds, so timestamps no longer depend on the server timezone.

All the documented columns are therefore required, the query has to be a
single SELECT, and a FORMAT clause is rejected with an explicit error.

Also spell out that a trace ID is 16 or 32 hexadecimal characters, which
SQL cannot match, with a test for a 16 character query.

Refs perses/perses#4202

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Erdinc Kaya <erdincka@msn.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Search output formatting, datasource variables, JSON attributes, service aggregation, and unbounded JSON-schema lookups need correction.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds ClickHouse trace lookup and search support for Perses trace panels.

Changes:

  • Implements trace retrieval, search aggregation, OTLP conversion, and editor UI.
  • Adds CUE schemas, Go SDK builders, documentation, and tests.
  • Registers and exposes ClickHouseTraceQuery.
File summaries
File Description
docs/clickhouse/README.md Introduces trace-query documentation.
docs/clickhouse/model.md Documents configuration and data model.
docs/clickhouse/go-sdk/trace-query.md Documents Go SDK usage.
clickhouse/src/queries/index.ts Exports the trace query.
clickhouse/src/queries/click-house-trace-query/index.ts Defines public exports.
clickhouse/src/queries/click-house-trace-query/get-click-house-trace-data.ts Implements lookup, search, and conversion.
clickhouse/src/queries/click-house-trace-query/get-click-house-trace-data.test.ts Tests trace data handling.
clickhouse/src/queries/click-house-trace-query/ClickHouseTraceQueryEditor.tsx Adds query options UI.
clickhouse/src/queries/click-house-trace-query/ClickHouseTraceQueryEditor.test.tsx Tests editor behavior.
clickhouse/src/queries/click-house-trace-query/ClickHouseTraceQuery.tsx Defines the plugin.
clickhouse/src/queries/click-house-trace-query/ClickHouseTraceQuery.test.ts Tests plugin options and dependencies.
clickhouse/src/queries/click-house-trace-query/click-house-trace-query-types.ts Defines configuration and row types.
clickhouse/sdk/go/query/trace/trace.go Adds the Go builder.
clickhouse/sdk/go/query/trace/trace_test.go Tests Go serialization.
clickhouse/sdk/go/query/trace/options.go Adds Go builder options.
clickhouse/schemas/queries/click-house-trace-query/query.cue Defines the CUE contract.
clickhouse/schemas/queries/click-house-trace-query/tests/valid/trace-id.json Covers valid trace lookup.
clickhouse/schemas/queries/click-house-trace-query/tests/valid/search.json Covers valid search configuration.
clickhouse/schemas/queries/click-house-trace-query/tests/invalid/non-positive-limit.json Rejects invalid limits.
clickhouse/schemas/queries/click-house-trace-query/tests/invalid/invalid-table.json Rejects unsafe table names.
clickhouse/schemas/queries/click-house-trace-query/tests/invalid/empty-query.json Rejects empty queries.
clickhouse/rsbuild.config.ts Exposes the plugin bundle.
clickhouse/package.json Registers the plugin metadata.
Review details
  • Files reviewed: 23/23 changed files
  • Comments generated: 5
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

WHERE TraceId != ''
GROUP BY TraceId
ORDER BY min(toDateTime64(Timestamp, 9)) DESC
LIMIT ${limit}`;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and worth fixing. The shared client appends FORMAT JSON only when the query doesn't contain the substring FORMAT (click-house-client.ts:60), so a search using formatDateTime(...) makes it skip that, ClickHouse answers TabSeparated, and response.json() throws. I'll append FORMAT JSON to the queries this plugin generates, so the client's check becomes a no-op for them, with a test covering formatDateTime in the subquery. The same substring check affects the existing log and time series queries, where the user's SQL is sent as-is; happy to open that separately as it touches the shared client.

Comment on lines +244 to +246
// The exporter stores every attribute value as a string, so the original value types cannot be recovered
function toKeyValues(attributes: Record<string, string> = {}): otlpcommonv1.KeyValue[] {
return Object.entries(attributes).map(([key, value]) => ({ key, value: { stringValue: value } }));

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and this PR was verified against the default Map schema. I'll make the attribute conversion total instead of assuming strings, mapping each JSON value to the matching OTLP AnyValue, and document the json: true schema. That also covers custom views returning non-string values.

Comment on lines +59 to +61
const client = (await context.datasourceStore.getDatasourceClient(
spec.datasource ?? DEFAULT_DATASOURCE,
)) as ClickHouseClient;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right that common.#datasourceSelector accepts datasource: "$var" and that this plugin passes the selector straight through. Worth noting it isn't a regression: ClickHouseLogQuery and ClickHouseTimeSeriesQuery behave identically, and the module's editors reject variable selections via isVariableDatasource, while Tempo and Jaeger use datasourceSelectValueToSelector. I'd rather fix the three ClickHouse queries together than make this one differ from its siblings — happy to do it as a follow-up, or here if you prefer.

Comment on lines +169 to +172
* The lookup is deliberately not bounded by time, so that a trace opens from a link whatever the dashboard time
* range. It relies on the bloom filter index the exporter creates on TraceId. The exporter's `<table>_trace_id_ts`
* table could bound the scan on large tables, but it only exists when the exporter created the schema. See "Trace ID
* lookup" in the data model docs.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the JSON schema: traces_json_table.sql has no TraceId bloom filter, only the attribute-key and duration indexes, so an unbounded lookup there can scan every partition. One correction:

_trace_id_ts and its materialized view are created in both modes — createTraceJSONTables calls renderCreateTraceIDTsTableSQL just as the Map path does — so the only case without it is a custom table or view. This is the decision flagged in the description. I lean towards deriving
_trace_id_ts and bounding the lookup by default, with an opt-out for custom tables. Happy to implement that if you agree.

Comment on lines +285 to +295
const serviceStats: Record<string, ServiceStats> = {};
for (const [serviceName, spanCount] of Object.entries(row.SpanCounts)) {
serviceStats[serviceName || 'unknown'] = { spanCount: Number(spanCount) };
}
for (const [serviceName, errorCount] of Object.entries(row.ErrorCounts)) {
const stats = serviceStats[serviceName || 'unknown'];
// ClickHouse also returns the services without errors, where the panels expect no count at all
if (stats !== undefined && Number(errorCount) > 0) {
stats.errorCount = Number(errorCount);
}
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and it came in with the SQL grouping change: an empty ServiceName and a real service named unknown normalise to the same key, and the second assignment overwrites the first in both loops. I'll accumulate, with a test for a trace that has both.

…service counts

Review feedback on the trace query, three fixes:

- Both generated queries end with an explicit FORMAT JSON. The shared
  client only appends it when the query doesn't contain the word FORMAT,
  which a search query may well do (formatDateTime), and ClickHouse would
  then answer in TabSeparated.
- Attribute values keep their type. The exporter's JSON schema (json:
  true) stores numbers, booleans and arrays, which were all wrapped in
  stringValue. They are now mapped to the matching OTLP value. ClickHouse's
  JSON type also turns the dots of attribute names into nesting, which is
  flattened back into the original names.
- The span and error counts of an empty service name and of a service
  named "unknown" are added rather than the last one overwriting the
  other, a regression from grouping the search in ClickHouse.

Refs perses/perses#4202

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Erdinc Kaya <erdincka@msn.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants