feat(tests): add tests for low and high cardinality request attribute…

[ArnabChatterjee20k]

What

Follow-up to #36 / #37, which added new request-attribute and premium-geo dimension columns to the events table. Those PRs created the columns as generic string attributes, so the ClickHouse adapter rendered them all as plain Nullable(String) with no index — the column existed but the data type, indexing, and compression weren't right for how they're queried. This PR picks the correct ClickHouse type, index, and codec per column, driven by the actual query surface and the FRA table's scale.

Query surface (why these choices)

Confirmed from the consuming PRs (appwrite-labs/cloud#5633, appwrite/appwrite#13463): all of these are used like existing console dimensions — selected, filtered (queries[]), and grouped-by (dimensions[]) — and filtered by exact equality (equal/in), not substring. Every query is primary-key-pruned to a (tenant, metric, time) slice first, so secondary indexes are an optimization on an already-small slice.

Schema — new columns

Column	Size	ClickHouse type	Index	Codec
protocol	16	LowCardinality(Nullable(String))	set(0)	—
accept	1024	Nullable(String)	bloom_filter	ZSTD(3)
acceptLanguage	256	Nullable(String)	bloom_filter	ZSTD(3)
queryKeys	1024	Nullable(String)	bloom_filter	ZSTD(3)
ipReputation (placeholder)	32	LowCardinality(Nullable(String))	set(0)	—
postalCode	32	Nullable(String)	bloom_filter	—
latitude	32	Nullable(String)	—	—
longitude	32	Nullable(String)	—	—
timeZone	64	LowCardinality(Nullable(String))	set(0)	—
weatherCode	16	LowCardinality(Nullable(String))	set(0)	—

Type rationale

• LowCardinality + set(0) for protocol / timeZone / weatherCode — intrinsically bounded value sets (protocol enum, ~400 IANA zones, code enum). Dictionary encoding + unlimited-set equality index are ideal and cheap.
• Nullable(String) + bloom_filter for accept / acceptLanguage / queryKeys — these store raw, un-normalized caller input (full Accept / Accept-Language headers, arbitrary query-key names), so their distinct count is unbounded. LowCardinality would risk dictionary/insert bloat at 6B rows; bloom_filter serves the exact-equality filters without a cardinality bet. ZSTD(3) compresses the high-entropy text (matching sibling city/isp).
• postalCode — high-cardinality, equality lookups → bloom_filter.
• latitude / longitude — display-only; nobody filters exact coordinates and a breakdown is meaningless → no index.
• ipReputation — placeholder, not populated yet. Holds a bounded IP-reputation verdict (clean / low / suspicious / block), exact-matched → LowCardinality + set(0), shaped correctly now so no second migration is needed when the producer is ready.

Indexes

• set(0): protocol, ipReputation, timeZone, weatherCode
• bloom_filter: accept, acceptLanguage, queryKeys, postalCode
• none: latitude, longitude
• accept / queryKeys (size 1024) carry a 255-char prefix (lengths) for the SQL adapter's 768-byte index-key limit; ClickHouse ignores the prefix and indexes the full value.

Index definitions live in Metric::getEventIndexes(). There is no auto-migration: createTable() emits them on fresh tables only. On the existing FRA table they are applied via a separately-run ALTER TABLE … ADD INDEX IF NOT EXISTS (metadata-only), and the ZSTD codec via ALTER TABLE … MODIFY COLUMN.

Storage cost (FRA: 6B+ rows, 1TB+)

Existing 6B rows were added via metadata-only ALTER ADD COLUMN, so they store NULL and cost ~nothing per value — the 1TB does not jump. Growth is incremental. Premium-geo is sparse (~20% of rows assumed); request attrs on ~100% of request rows.

Columns — per 1B new rows (~29 GB total): accept ~13, queryKeys ~8, acceptLanguage ~4.5 (all ZSTD(3) — saves ~20 GB/1B vs LZ4), postalCode+lat+long ~3.4, protocol/timeZone/weatherCode ~0.3.

Indexes — ~8–9 GB @ 6B (~1.4 GB/1B): dominated by the queryKeys bloom (~5.5 GB); accrues only on new/merged parts unless MATERIALIZE INDEX is run over history.

ipReputation is unpopulated today → stores NULL, costs ~nothing (its populated cost is added separately below, since it lands later).

	One-time (6B history)	Per +1B new rows
Columns	<2 GB (null-map)	~29 GB
Indexes	~8–9 GB (only if backfilled; else 0)	~1.4 GB
Total	~8–11 GB	~30 GB / 1B

Later: when ipReputation is populated (added on top of the total above)

ipReputation ships as an unpopulated placeholder, so it contributes nothing until the producer is wired up. Once it carries values it's a ~5-value verdict enum (clean/low/suspicious/block + unknown):

	per 1B rows	@ 6B rows (fully populated)
Column (LowCardinality)	~0.05–0.1 GB	~0.3–0.6 GB
set(0) index (≤5 entries/granule)	~0.02 GB	~0.1–0.2 GB
Added total	~0.1 GB	~0.5–0.8 GB

Effectively free — a 5-value dictionary compresses the reference stream to well under 0.1 B/row, and the per-granule set holds at most the handful of distinct verdicts. Even at full 6B-row coverage it adds well under 1 GB on top of the totals above.

Bottom line

The FRA 1 TB does not jump — history stays NULL (~free). Going 6B → 8B ≈ +0.06 TB (~60 GB). The design is the cardinality-safe option: LowCardinality only where value sets are genuinely bounded, bloom_filter + ZSTD(3) for un-normalized text (equality-serving, no cardinality bet), and no wasted indexes on display-only coordinates. ALTER ADD COLUMN / ADD INDEX are metadata-only and safe at this scale; the only heavy operation (MATERIALIZE INDEX over history) is left as an explicit, out-of-band ops choice.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QssJBktSLWpuEuWzVB23do

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because no new blocking correctness or security failure remains.

Summary

• Uses low-cardinality strings and set indexes for bounded dimensions.
• Keeps unbounded request text as nullable strings with bloom filters and ZSTD compression.
• Leaves display-only coordinates unindexed.
• Adds schema and metric tests for the resulting configuration.

_{Reviews (8) · Last reviewed commit: "feat(metric): add ipReputation placehold..."}