feat(endpoint): add Scalar schema for lens-dependent entity fields by ntucker · Pull Request #3887 · reactive/data-client

ntucker · 2026-04-08T03:56:57Z

Motivation

A primary use case is displaying vast amounts of information in a grid (like a spreadsheet). Each row is an entity, but some columns display relational data that depends on a user selection — for example, which portfolio's equity percentages to show for each company. We call this selection a "lens."

The goal is to:

Cache different lens selections independently so switching back doesn't require a refetch
Support column-only fetches that update just the lens data without touching entity fields
Allow multiple components to render the same entity through different lenses simultaneously

Solution

Introduces a Scalar schema for lens-dependent entity fields.

Normalize: Entity fields like pct_equity: 0.5 are replaced in the entity row with lens-independent wrappers { id, field, entityKey }. The actual scalar data is stored in a Scalar(<key>) entity table keyed by entityKey|entityPk|lensValue.

Denormalize: EntityMixin.denormalize is unchanged. The standard unvisit(schema, input[key]) loop calls Scalar.denormalize, which reads the wrapper, computes the lens from current endpoint args, looks up the correct cell, and returns just the field value.

const PortfolioScalar = new Scalar({
  lens: args => args[0]?.portfolio,
  key: 'portfolio',
  entity: Company,
});

class Company extends Entity {
  id = '';
  price = 0;
  pct_equity = 0;
  shares = 0;
  static schema = {
    pct_equity: PortfolioScalar,
    shares: PortfolioScalar,
  };
}

// Column-only endpoint (dictionary keyed by company pk)
const getPortfolioColumns = new Endpoint(
  ({ portfolio }) => fetch(`/companies/columns?portfolio=${portfolio}`),
  { schema: new schema.Values(PortfolioScalar) },
);

Design iteration

Started with a two-class design (Scalar + internal ScalarCell) so the cell could be entity-like (have pk()) for storage in the entity table, while Scalar stayed lens-aware. The ScalarCell was a pass-through that existed solely to satisfy normalizr's entity-shape constraints.

Collapsed to a single class. Scalar now implements Mergeable directly (provides merge/mergeWithStore/mergeMetaWithStore/shouldReorder/createIfValid) and stores cell data under a compound pk (entityKey|entityPk|lensValue). The unvisit.ts denormalize router learned a small branch to route schemas with .key and string inputs through unvisitEntity even when they don't expose .pk — letting Scalar be table-resident without pretending to be an Entity.

Generalized parent-entity context to the visit walker. Earlier the entity context was smuggled in by passing the whole entity as parent and encoding <entityKey>|<entityPk>|<fieldName> into the key arg, breaking the standard Visit contract. The visit walker (getVisit) now tracks the nearest enclosing entity-like schema (anything with pk) in a closure variable and passes it as a 7th parentEntity argument to every schema.normalize call. A new acceptsPrimitives opt-in marker lets schemas like Scalar receive primitive field values instead of being short-circuited. With that in place, EntityMixin.normalize's field loop is one uniform visit(...) — no Scalar-specific branch — and Scalar.normalize reads parentEntity.key / parentEntity.pk(parent, …, args) to derive the cell's compound key. parent is the entity data row as the contract specifies.

Performance investigation

Validated with 8-run A/B benchmarks (yarn start normalizr "^normalize"), stash-pop sequence to keep system state comparable across A and B.

Approach	`normalizeLong`	`normalizeLong Values`
Baseline (HEAD before this PR)	770 ops/sec	690 ops/sec
`delegate.currentEntity` mutation (per-entity write)	—	−3 to −4%
`Object.defineProperty` getter on `delegate`	—	−44% (catastrophic)
Dispatch in `EntityMixin` (one `acceptsPrimitives` check, no walker change)	770 (0%)	686 (−0.6%, within noise)
Final: closure tracking in `getVisit` + 7th arg + `acceptsPrimitives`	741 (−3.8%)	660 (−4.3%)

Findings:

The delegate mutation regression came from V8 IC pollution: assigning different Entity subclasses (each with a distinct hidden class) to a single property turned the IC megamorphic and propagated deopts into hot inlined call sites (normalize, Object_normalize, normalizeValue). Confirmed with --trace-deopt.
The Object.defineProperty getter regression was due to installing an accessor property on delegate — V8 deoptimizes other property accesses on the object once any property becomes an accessor.
The closure save/restore around every entity visit (prev = currentEntity; currentEntity = schema; …; currentEntity = prev) is not a deopt — it's steady-state work in fully optimized code. --trace-deopt was clean and gave a misleading green light at first; only a tight A/B benchmark caught it. Lesson: deopt traces tell you whether you broke optimization, but only A/B benchmarks tell you whether the optimized version is fast enough.
Passing an extra 7th arg to schema.normalize is essentially free in modern V8.

Trade-off chosen: ~3% normalize-throughput cost on the hot path in exchange for a uniform schema contract. EntityMixin has no Scalar-specific branch; any future context-dependent schema (not just Scalar) can read its enclosing entity from the standard 7th arg.

Files touched

packages/endpoint/src/schemas/Scalar.ts — new schema
packages/endpoint/src/schemas/EntityMixin.ts — field-normalize loop simplified to a single visit call
packages/normalizr/src/denormalize/unvisit.ts — route table-resident schemas with .key (no .pk) through unvisitEntity
packages/normalizr/src/normalize/getVisit.ts — track currentEntity, pass as 7th arg, honor acceptsPrimitives
packages/normalizr/src/interface.ts — SchemaSimple.normalize declares the optional parentEntity arg
packages/endpoint/src/schemas/__tests__/Scalar.test.ts — coverage
docs/rest/api/Scalar.md — API docs

Open questions

Should Scalar support a process or validate hook for cell data?
Should the compound-pk delimiter (|) be configurable for entity keys/pks that legitimately contain |?
MemoCache caching behavior with different args for the same entity — currently requires separate memo instances per lens. Integration with useSuspense / useQuery hook args should handle this naturally.

changeset-bot · 2026-04-08T03:57:02Z

🦋 Changeset detected

Latest commit: a9e969c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@data-client/endpoint	Major
@data-client/rest	Major
@data-client/graphql	Major
@data-client/normalizr	Major
@data-client/core	Major
@data-client/react	Major
@data-client/vue	Minor
@data-client/img	Major
example-benchmark	Patch
example-benchmark-react	Patch
normalizr-github-example	Patch
normalizr-redux-example	Patch
normalizr-relationships	Patch
test-bundlesize	Patch
coinbase-lite	Patch
@data-client/test	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-04-08T03:57:05Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs-site	Ignored	Preview	Apr 22, 2026 1:47pm

github-actions · 2026-04-08T03:59:06Z

Size Change: +23 B (+0.03%)

Total Size: 80.7 kB

📦 View Changed

Filename	Size	Change
`examples/test-bundlesize/dist/rdcClient.js`	10.5 kB	+23 B (+0.22%)

ℹ️ View Unchanged

Filename	Size
`examples/test-bundlesize/dist/App.js`	1.46 kB
`examples/test-bundlesize/dist/polyfill.js`	307 B
`examples/test-bundlesize/dist/rdcEndpoint.js`	8.01 kB
`examples/test-bundlesize/dist/react.js`	59.7 kB
`examples/test-bundlesize/dist/webpack-runtime.js`	726 B

_{compressed-size-action}

codecov · 2026-04-08T04:00:10Z

Codecov Report

❌ Patch coverage is 98.18182% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 98.13%. Comparing base (714348e) to head (0c3e870).

Files with missing lines	Patch %	Lines
packages/core/src/controller/Controller.ts	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3887      +/-   ##
==========================================
+ Coverage   98.11%   98.13%   +0.02%     
==========================================
  Files         153      154       +1     
  Lines        2917     2955      +38     
  Branches      567      581      +14     
==========================================
+ Hits         2862     2900      +38     
  Misses         11       11              
  Partials       44       44

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions

Benchmark

Details

Benchmark suite	Current: `a9e969c`	Previous: `4f5b6b6`	Ratio
`normalizeLong`	`458` ops/sec (`±2.27%`)	`458` ops/sec (`±1.29%`)	`1`
`normalizeLong Values`	`416` ops/sec (`±1.27%`)	`417` ops/sec (`±0.37%`)	`1.00`
`denormalizeLong`	`247` ops/sec (`±4.11%`)	`394` ops/sec (`±1.17%`)	`1.60`
`denormalizeLong Values`	`225` ops/sec (`±3.96%`)	`339` ops/sec (`±0.50%`)	`1.51`
`denormalizeLong donotcache`	`1029` ops/sec (`±0.12%`)	`1015` ops/sec (`±0.10%`)	`0.99`
`denormalizeLong Values donotcache`	`749` ops/sec (`±0.19%`)	`758` ops/sec (`±0.23%`)	`1.01`
`denormalizeShort donotcache 500x`	`1427` ops/sec (`±0.21%`)	`1558` ops/sec (`±0.14%`)	`1.09`
`denormalizeShort 500x`	`690` ops/sec (`±3.81%`)	`1060` ops/sec (`±2.16%`)	`1.54`
`denormalizeShort 500x withCache`	`5670` ops/sec (`±0.28%`)	`7687` ops/sec (`±0.11%`)	`1.36`
`queryShort 500x withCache`	`2884` ops/sec (`±0.55%`)	`2999` ops/sec (`±0.11%`)	`1.04`
`buildQueryKey All`	`58539` ops/sec (`±0.48%`)	`54049` ops/sec (`±0.42%`)	`0.92`
`query All withCache`	`5103` ops/sec (`±0.20%`)	`6752` ops/sec (`±0.32%`)	`1.32`
`denormalizeLong with mixin Entity`	`238` ops/sec (`±4.18%`)	`357` ops/sec (`±2.60%`)	`1.50`
`denormalizeLong withCache`	`8115` ops/sec (`±0.39%`)	`7722` ops/sec (`±0.45%`)	`0.95`
`denormalizeLong Values withCache`	`4622` ops/sec (`±0.45%`)	`5215` ops/sec (`±0.19%`)	`1.13`
`denormalizeLong All withCache`	`4463` ops/sec (`±0.17%`)	`6457` ops/sec (`±0.11%`)	`1.45`
`denormalizeLong Query-sorted withCache`	`5196` ops/sec (`±0.21%`)	`6764` ops/sec (`±0.21%`)	`1.30`
`denormalizeLongAndShort withEntityCacheOnly`	`1845` ops/sec (`±0.24%`)	`1837` ops/sec (`±0.18%`)	`1.00`
`denormalize bidirectional 50`	`4815` ops/sec (`±4.17%`)	`6952` ops/sec (`±0.28%`)	`1.44`
`denormalize bidirectional 50 donotcache`	`40073` ops/sec (`±1.66%`)	`40359` ops/sec (`±1.32%`)	`1.01`
`getResponse`	`3757` ops/sec (`±0.47%`)	`4714` ops/sec (`±0.69%`)	`1.25`
`getResponse (null)`	`9921715` ops/sec (`±0.30%`)	`10852221` ops/sec (`±0.75%`)	`1.09`
`getResponse (clear cache)`	`232` ops/sec (`±4.23%`)	`338` ops/sec (`±0.33%`)	`1.46`
`getSmallResponse`	`3506` ops/sec (`±0.23%`)	`3511` ops/sec (`±0.09%`)	`1.00`
`getSmallInferredResponse`	`2779` ops/sec (`±0.15%`)	`2617` ops/sec (`±0.11%`)	`0.94`
`getResponse Collection`	`3774` ops/sec (`±0.29%`)	`4708` ops/sec (`±0.82%`)	`1.25`
`get Collection`	`3557` ops/sec (`±0.19%`)	`4732` ops/sec (`±0.47%`)	`1.33`
`get Query-sorted`	`5037` ops/sec (`±0.25%`)	`4672` ops/sec (`±0.27%`)	`0.93`
`setLong`	`465` ops/sec (`±0.51%`)	`469` ops/sec (`±0.13%`)	`1.01`
`setLongWithMerge`	`259` ops/sec (`±0.21%`)	`255` ops/sec (`±0.21%`)	`0.98`
`setLongWithSimpleMerge`	`277` ops/sec (`±0.19%`)	`277` ops/sec (`±0.12%`)	`1`
`setSmallResponse 500x`	`909` ops/sec (`±0.93%`)	`918` ops/sec (`±0.74%`)	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Benchmark React

Details

Benchmark suite	Current: `a9e969c`	Previous: `ce99359`	Ratio
`data-client: getlist-100`	`139.87` ops/s (`± 5.4%`)	`136.99` ops/s (`± 5.1%`)	`0.98`
`data-client: getlist-500`	`43.1` ops/s (`± 5.4%`)	`41.85` ops/s (`± 4.7%`)	`0.97`
`data-client: update-entity`	`416.67` ops/s (`± 10.0%`)	`333.33` ops/s (`± 9.8%`)	`0.80`
`data-client: update-user`	`363.76` ops/s (`± 9.7%`)	`344.83` ops/s (`± 7.8%`)	`0.95`
`data-client: getlist-500-sorted`	`43.49` ops/s (`± 6.4%`)	`46.08` ops/s (`± 6.6%`)	`1.06`
`data-client: update-entity-sorted`	`384.62` ops/s (`± 5.8%`)	`294.12` ops/s (`± 7.4%`)	`0.76`
`data-client: update-entity-multi-view`	`344.83` ops/s (`± 9.3%`)	`317.54` ops/s (`± 8.1%`)	`0.92`
`data-client: list-detail-switch-10`	`8.18` ops/s (`± 7.1%`)	`7.48` ops/s (`± 6.1%`)	`0.91`
`data-client: update-user-10000`	`79.05` ops/s (`± 15.4%`)	`74.91` ops/s (`± 14.1%`)	`0.95`
`data-client: invalidate-and-resolve`	`36.36` ops/s (`± 5.0%`)	`36.1` ops/s (`± 5.2%`)	`0.99`
`data-client: unshift-item`	`232.56` ops/s (`± 3.8%`)	`238.1` ops/s (`± 2.1%`)	`1.02`
`data-client: delete-item`	`285.71` ops/s (`± 3.8%`)	`270.27` ops/s (`± 7.0%`)	`0.95`
`data-client: move-item`	`194.19` ops/s (`± 6.7%`)	`181.82` ops/s (`± 5.0%`)	`0.94`

This comment was automatically generated by workflow using github-action-benchmark.

Introduces Scalar + ScalarCell classes following the Union pattern: - Scalar (SchemaSimple, not entity-like) routes normalize/denormalize - ScalarCell (entity-like, internal) stores grouped cell data - EntityMixin.normalize: if/else to pass whole entity to Scalar - EntityMixin.denormalize: completely unchanged (Union-like wrapper) - Entity stores lens-independent {id,field} wrappers - Denormalize joins correct cell based on endpoint args Co-authored-by: natmaster <natmaster@gmail.com>

Tests cover: - normalize stores wrapper refs in entity, cell data in ScalarCell - multiple entities, different lenses produce separate cells - denormalize joins correct cell based on lens args - different lens args produce different results from same entity - missing lens returns undefined for scalar fields - column-only fetch via Values stores cells without modifying entities - column fetch cells joinable via denormalize with Company schema - merge accumulation updates existing cells - Scalar constructor and queryKey Co-authored-by: natmaster <natmaster@gmail.com>

Covers usage (full entity + column-only endpoints), constructor options, normalize/denormalize flow, normalized storage model, and related APIs. Co-authored-by: natmaster <natmaster@gmail.com>

_lastCpk was declared and initialized but never read or written elsewhere in the codebase. Co-authored-by: Nathaniel Tucker <me@ntucker.me>

Replace the encoded-key hack with a direct `parentEntity` argument: - `EntityMixin.normalize` now dispatches schemas marked `acceptsPrimitives` directly (bypassing `visit`'s primitive short-circuit) and passes `this` as the 7th arg. - `Scalar.normalize` reads `parentEntity` to derive entity key and pk; no more parsing `'<entityKey>|<entityPk>|<fieldName>'` out of the visit key. - `parent` is now the entity data row (standard `Visit` contract), not the Entity class. - `getVisit` and the `SchemaSimple` interface are unchanged — zero impact on the normalize hot path (verified at parity with HEAD across 8-run A/B benchmarks). Made-with: Cursor

Move parent-entity context tracking into `getVisit` itself, eliminating the per-schema-type dispatch in `EntityMixin`. The walker now: - Maintains a `currentEntity` closure variable, save/restored around every entity visit (schemas with `pk`). - Passes it as a 7th `parentEntity` arg to every `schema.normalize` call. - Honors a new `acceptsPrimitives` opt-in marker so schemas like `Scalar` receive primitive values instead of being short-circuited. `EntityMixin.normalize`'s field loop is now a single uniform `visit(...)` call — no more conditional branch for Scalar fields. `Scalar.normalize` reads `parentEntity` from the standard 7th arg; `parent` is the entity data row as the standard `Visit` contract specifies. Trade-off: ~3% normalize-throughput cost on the hot path (closure save/restore around every entity visit). Validated with 8-run A/B benchmarks. Buys a uniform schema contract — Scalar (and any future context-dependent schema) needs no special case in `EntityMixin`. Made-with: Cursor

Both branches called `schema.normalize` with the same args except for the parent-entity context. Snapshot `prev = currentEntity` first, then conditionally update `currentEntity = schema` for entities. Pass `prev` — which equals the prior entity for entities and the still-current entity for non-entities — and unconditionally restore. One call site instead of two, no behavior change. 8-run A/B benchmarks at parity with the prior version (within noise). Made-with: Cursor

The `else if` branch for table-resident schemas without `pk` was matching any schema exposing a `key` property. `Invalidate` extends PolymorphicSchema and exposes `key` via a getter, so it was being routed into `unvisitEntity` -> `unvisitEntityObject`, which calls `schema.createIfValid()` -- a method `Invalidate` does not implement. This caused `TypeError` on basic Invalidate denormalization and broke deletion/symbol propagation. Switch the discriminator to `typeof createIfValid === 'function'`, which is the precise capability `unvisitEntityObject` requires. This matches Scalar (which now implements Mergeable) and regular entities, but not Invalidate, Query, Union, etc. -- they continue falling through to their own `denormalize` methods. Made-with: Cursor

Add direct tests for merge/shouldReorder/mergeWithStore/mergeMetaWithStore and denormalize passthroughs (falsy, symbol, object, missing-lens, missing cell, cpk string + Values round-trip) to bring Scalar.ts to 100% coverage. Made-with: Cursor

`getResponseMeta` short-circuits `denormalize()` when the endpoint schema doesn't transform the response. The previous check (`schemaHasEntity`) had two bugs surfaced by Scalar: 1. For `Values(Scalar)` it walked `Object.values(scalarInstance)`, recursed into the `lensSelector` function, and looped forever. 2. Scalar is table-resident without `pk`, so it was not detected as needing denormalize — the raw cpk strings were returned instead of the joined cell data. Replace with `requiresDenormalize`, which mirrors `getVisit.ts`: schemas that define `normalize` always need denormalize to reconstruct the response. This collapses the entity check, the Scalar special- case, the wrapper `.schema` recursion, and the self-loop guard into a single early-exit, so schema class instances never fall through to `Object.values()` traversal of their instance fields. Adds regression tests for both `Values(Scalar)` (column-only endpoint) and Entity-with-Scalar-fields, with a hard 2s timeout so a re- introduced infinite recursion fails fast rather than hanging Jest. Net: -89B minified, neutral-to-positive on `core ^get` benchmarks. Made-with: Cursor

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a9e969c. Configure here.}

cursor · 2026-04-22T13:58:36Z

-    }
-    return Object.values(nestedSchema).some(x => schemaHasEntity(x));
-  }
-  return false;


requiresDenormalize matches strings via String.prototype.normalize

Low Severity

The requiresDenormalize function uses typeof (schema as any).normalize === 'function' as its primary check. When recursing through Object.values(...) of a plain-object schema, if any value happens to be a string, String.prototype.normalize satisfies this check and incorrectly returns true. The old schemaHasEntity used isEntity() (checks .pk) which never matched primitives. While plain-object schemas rarely contain string values in practice, this is a behavioral regression that could cause unnecessary denormalization work.

^{Reviewed by Cursor Bugbot for commit a9e969c. Configure here.}

github-actions Bot reviewed Apr 8, 2026

View reviewed changes

ntucker marked this pull request as ready for review April 8, 2026 05:40

cursor Bot reviewed Apr 8, 2026

View reviewed changes

Comment thread packages/endpoint/src/schemas/Scalar.ts Outdated

Comment thread packages/endpoint/src/schemas/Scalar.ts Outdated

Comment thread packages/endpoint/src/schemas/Scalar.ts Outdated

ntucker force-pushed the cursor/scalar-schema-design-for-lenses-d6e3 branch from 352ef6f to f989a17 Compare April 13, 2026 13:04

vercel Bot deployed to Preview April 17, 2026 00:25 View deployment