diff --git a/docs/foomatic-data-formats.md b/docs/foomatic-data-formats.md new file mode 100644 index 00000000..1ccb063c --- /dev/null +++ b/docs/foomatic-data-formats.md @@ -0,0 +1,205 @@ +# Foomatic Pipeline — Data Formats + +This document is the schema reference for every JSON artifact produced by the pipeline described in [foomatic-pipeline-architecture.md](./foomatic-pipeline-architecture.md). All artifacts live under `public/foomatic-db/` and are served as static files — none require a backend to read — except the two build-time intermediates, `feature-matrix.json` and `recommendations.json`, which are written to the git-ignored `cache/foomatic/` and are not part of the static export. + +The canonical TypeScript types are defined in [`lib/foomatic/types.ts`](../lib/foomatic/types.ts); this document explains what each field means and which pipeline stage produces it. + +--- + +## `public/foomatic-db/printer/*.json`, `driver/*.json` + +**Produced by:** `generate-from-xml.ts` +**Consumed by:** `combine-data.ts` + +A 1:1 JSON mirror of the upstream Foomatic XML, one file per `` or `` element. Shape is whatever `fast-xml-parser` produces from the source XML (attributes prefixed with `@`, text nodes under `#text`), with one normalization: the `` tag is renamed to `driverPrototype`. These are intermediate artifacts — nothing in the frontend reads them directly. + +--- + +## `public/foomatic-db/printers.json` + +**Produced by:** `combine-data.ts` +**Consumed by:** `split-printers.ts`, `vectorize.ts`, `compute-similarity.ts` + +The unified, normalized dataset — one record per printer, each conforming to the `Printer` interface: + +```ts +interface Printer { + id: string // normalized id, e.g. "HP-LaserJet-4" + manufacturer: string + model: string + series?: string + connectivity?: string[] // ["USB", "Network", ...] + recommended_driver?: string // e.g. "driver/Postscript-hp" + drivers?: Driver[] + type?: string // "inkjet" | "laser" | "dot-matrix" | "unknown" + status?: string // "Perfect" | "Mostly" | "Unsupported" | "Unknown" + // derived from `functionality`; falls to "Unsupported" + // when no usable driver remains (see below) + notes?: string // HTML from upstream XML — sanitize before rendering + functionality?: string // raw Foomatic grade: "A" | "B" | "C" | "?" + commandsets?: string[] // normalized PDL tokens, e.g. ["PCLXL", "POSTSCRIPT"] + ppdOptions?: PpdOption[] + color?: boolean | "unknown" + duplex?: boolean | "unknown" + recommended?: boolean + psLevel?: number | null // 0–3 + pclLevel?: number | null // 0, 3, 4, 5, or 6 + maxDpi?: number | null +} + +interface Driver { + id: string + name: string + url?: string + comments?: string // HTML from upstream XML — sanitize before rendering + hasPpd?: boolean + ppdPath?: string + execution?: { ghostscript?: string | null; filter?: string | null; prototype: string } +} +``` + +`notes` and `driver.comments` are sourced from upstream contributor-editable XML and **must** be passed through `sanitizeFoomaticHtml()` (`lib/foomatic/sanitize.ts`) before being rendered with `dangerouslySetInnerHTML` — see that file's usage in `components/foomatic/PrinterPageClient.tsx`. + +--- + +## `public/foomatic-db/printersMap.json` + +**Produced by:** `split-printers.ts` +**Consumed by:** the directory listing page (`app/foomatic/printers/page.tsx`) and `generateStaticParams()` in `app/foomatic/printer/[make]/[id]/page.tsx` + +A lightweight projection of `printers.json`, one entry per printer, used so the directory page doesn't need to download every printer's full driver/PPD detail just to render a list: + +```ts +interface PrinterSummary { + id: string + manufacturer: string + model: string + type?: string + status?: string + driverCount?: number + functionality?: string +} +``` + +`driverCount` is `drivers.length` from the full record; `type`/`status`/`functionality` default to `"unknown"`/`"Unknown"`/`"?"` respectively if absent. + +### Usable drivers and support status + +A driver entry carries `obsolete` and, when upstream names a successor, `replacedBy`. +An obsolete driver is still listed and badged on the printer and driver pages, but it +is not usable: it contributes no compatibility evidence to the recommendation engine, +and it does not count towards the printer having driver support. + +`status` therefore falls to `"Unsupported"` — the site's existing representation of the +unsupported/"paperweight" level — when a printer has no usable driver **and** its +recorded `functionality` grade is unknown. A recorded grade is never overwritten: a +printer graded A or B keeps `"Perfect"`/`"Mostly"` even if every listed driver is +obsolete, because that grade reflects observed behaviour rather than driver +availability. + +`printersMap.json` carries a driver total but not each driver's `obsolete` flag, so a +summary cannot recompute this itself; `calculateAccurateStatus()` counts usable drivers +when given a full record and otherwise defers to the `status` the pipeline stored. + +--- + +## `public/foomatic-db/printers/.json` + +**Produced by:** `split-printers.ts` +**Consumed by:** the printer detail page (`PrinterPageClient.tsx`) + +One full `Printer` record per printer (same shape as an entry in `printers.json`), written individually so a detail-page visit only fetches data for that one printer. + +--- + +## `cache/foomatic/feature-matrix.json` + +**Produced by:** `vectorize.ts` +**Consumed by:** `compute-similarity.ts` + +```ts +interface FeatureMatrix { + printerCount: number + featureCount: number + featureNames: string[] // e.g. ["recommended_driver:postscript", ..., "res:2400plus"] + vocab: { + recommendedDrivers: string[] + supportedDrivers: string[] + types: string[] + commandsets: string[] // only commandsets meeting MIN_COMMANDSET_FREQUENCY (20) + } + ids: string[] // printer ids, parallel-indexed with matrix rows + matrix: number[][] // one weighted feature vector per printer +} +``` + +`matrix[i]` corresponds to `ids[i]` and is encoded in the exact order of `featureNames`. See [foomatic-recommendation-quality.md](./foomatic-recommendation-quality.md) for what each feature means and its weight. + +--- + +## `cache/foomatic/recommendations.json` + +**Produced by:** `compute-similarity.ts` +**Consumed by:** dev/debug tooling and as the source for the per-printer split below (not fetched directly by the frontend — see `recommendations/.json`) + +```ts +interface Output { + version: string // "2.0.0" + printerCount: number + topK: number // 10 + recommendations: { + [printerId: string]: Array<{ + id: string // recommended printer's id + score: number // engineered similarity in [0, 1): IDF-weighted + // cosine x evidence damping x conflict penalties, + // rounded to 3 decimals — see the Scoring Model + // section of foomatic-recommendation-quality.md + sharedFeatures: string[] // human-readable explanation strings + }> + } +} +``` + +The score is deterministic and bounded. It is **not** a probability, not a +guarantee that a printer will work, and not a human-validated accuracy figure — +it summarizes how much recorded Foomatic evidence two printers share, minus +penalties for conflicting capabilities. The UI maps it onto four tiers defined +in `lib/foomatic/scoring.ts` (`confidenceTier()`). + +`sharedFeatures` entries are generated by `computeSharedFeatures()` and look like `"Shared command set: PostScript"`, `"Color printing"`, `"1200 dpi resolution"`, `"Excellent Linux driver support"` — these are rendered directly as the "why this printer?" list on the printer detail page. + +--- + +## `public/foomatic-db/recommendations/.json` + +**Produced by:** `compute-similarity.ts` (same run as above, written per-printer for fetch efficiency) +**Consumed by:** `RecommendedPrintersSection.tsx` + +A single printer's recommendation array — `recommendations.json`'s `recommendations[id]` value, plus the display fields each card needs: + +```ts +type RecommendationsForPrinter = Array<{ + id: string + score: number + sharedFeatures: string[] + // Denormalized from printers.json so the printer page can render the + // recommendation cards from this one file alone. `status` and `type` + // defaults mirror the printersMap.json projection. + manufacturer?: string + model?: string + status: string // "Perfect" | "Mostly" | ... | "Unknown" + type: string // "laser" | "inkjet" | "dot-matrix" | "unknown" +}> +``` + +The detail page only ever needs the current printer's own recommendations, so fetching this file instead of the ~23 MB combined `recommendations.json` is what keeps the printer detail page's initial load small (see commit `perf(recommendations): split recommendation data per printer`). + +A driver count is deliberately **not** among these fields. `printersMap.json` still carries `driverCount` for the directory listing and search index, but a recommendation card must not be able to render it: the number of driver entries a model accumulates upstream is not a measure of how well it is supported (see [foomatic-recommendation-quality.md](./foomatic-recommendation-quality.md#driver-count-is-not-a-support-quality-signal)). + +The denormalized display fields cost roughly 0.8 KB per shard (median 3.3 KB at the current snapshot) but remove a second ~1.5 MB `printersMap.json` fetch that the section previously needed purely to resolve manufacturer/model/status for the three cards it renders — a net reduction from ~1,495 KB to ~3 KB per printer-page visit. They are intentionally *not* added to the combined `recommendations.json`, which stays a compact diagnostic artifact. + +--- + +## Versioning + +`recommendations.json` carries an explicit `version` field (`"2.0.0"` as of the resolution-aware similarity feature addition). There is no compatibility-checking consumer of this field today — it exists as a marker for pipeline output changes across the feature-addition history documented in [foomatic-recommendation-quality.md](./foomatic-recommendation-quality.md). If the feature vector schema changes again (a new weighted dimension, a changed weight, or a different scoring formula), bump this version so it's traceable in the artifact itself. diff --git a/docs/foomatic-pipeline-architecture.md b/docs/foomatic-pipeline-architecture.md new file mode 100644 index 00000000..b937b5fe --- /dev/null +++ b/docs/foomatic-pipeline-architecture.md @@ -0,0 +1,192 @@ +# Foomatic Recommendation Pipeline — Architecture + +## Overview + +This document describes the offline, reproducible recommendation pipeline that turns the upstream [OpenPrinting/foomatic-db](https://github.com/OpenPrinting/foomatic-db) XML dataset into the static printer directory, printer detail pages, and "similar printers" recommendations served from `openprinting.github.io/foomatic`. + +The pipeline is designed to: + +- Run entirely offline, with no server-side inference at request time +- Produce static, versioned JSON artifacts consumable by a static-exported Next.js site (no backend) +- Be fully reproducible from upstream Foomatic XML on every run +- Refresh automatically via GitHub Actions as the upstream database changes + +This is the GSoC "Track A" deliverable: *an offline pipeline for printer similarity and compatibility analysis, exported as static artifacts for static-site consumption.* + +--- + +## High-Level Architecture + +The pipeline is divided into seven sequential stages, each implemented as an independent `tsx` script under `scripts/foomatic/`, orchestrated by `scripts/foomatic/data-generate.ts`: + +``` +1. generate-from-xml.ts XML → JSON ingestion +2. generate-ppds.sh PPD compilation (Linux only) +3. combine-data.ts Normalization + enrichment +4. split-printers.ts Performance: per-printer artifact split +5. split-drivers.ts Performance: per-driver artifact split +6. vectorize.ts Feature engineering +7. compute-similarity.ts Similarity computation + recommendations +``` + +Each stage reads the previous stage's output from `public/foomatic-db/` and writes its own output back to the same directory (the two build-time intermediates, the feature matrix and the full recommendation map, go to `cache/foomatic/` instead), so the pipeline can be re-run incrementally or in full. Stage failure is fail-fast: if any step exits non-zero, the orchestrator stops immediately rather than continuing with stale or partial data. + +--- + +## Pipeline Flow + +``` + OpenPrinting/foomatic-db (git, upstream) + │ + │ git clone / pull + ▼ + cache/foomatic-db/db/source/{printer,driver}/*.xml + │ + ┌─────────────┴─────────────┐ + │ generate-from-xml.ts │ fast-xml-parser + └─────────────┬─────────────┘ + ▼ + public/foomatic-db/printer/*.json driver/*.json + │ + ┌─────────────┴─────────────┐ + │ generate-ppds.sh │ foomatic-compiledb + │ (skipped on Windows / │ (cached by git revision) + │ --skip-ppd) │ + └─────────────┬─────────────┘ + ▼ + public/ppds/*.ppd + │ + ┌─────────────┴─────────────┐ + │ combine-data.ts │ joins printer ⇄ driver + │ │ graphs, derives all + │ │ normalized attributes + └─────────────┬─────────────┘ + ▼ + public/foomatic-db/printers.json + (the "unified and normalized printer + metadata dataset" deliverable) + │ + ┌─────────────┴─────────────┐ + │ split-printers.ts │ performance split + └─────────────┬─────────────┘ + ▼ + public/foomatic-db/printersMap.json printers/.json + │ + ┌─────────────┴─────────────┐ + │ vectorize.ts │ builds vocabularies, + │ │ encodes weighted + │ │ feature vectors + └─────────────┬─────────────┘ + ▼ + cache/foomatic/feature-matrix.json + │ + ┌─────────────┴─────────────┐ + │ compute-similarity.ts │ IDF-weighted similarity, + │ │ evidence + conflict + │ │ scoring, top-10 + └─────────────┬─────────────┘ + ▼ + cache/foomatic/recommendations.json (full map, not exported) + public/foomatic-db/recommendations/.json (per-printer) + │ + ▼ + Next.js static export (output: "export") + app/foomatic/printers/page.tsx · app/foomatic/printer/[make]/[id]/page.tsx + (fetched client-side, zero backend, per the + Track B static-site requirement) +``` + +--- + +## Stage Details + +### 1. XML Ingestion — `generate-from-xml.ts` + +- Clones `OpenPrinting/foomatic-db` into `cache/foomatic-db` on first run, or `git pull`s it on subsequent runs (a failed pull is logged but non-fatal, so a stale local cache doesn't block the rest of the pipeline). +- Parses every `printer/*.xml` and `driver/*.xml` file with `fast-xml-parser`, configured to preserve XML attributes (`@`-prefixed) and rename the `` tag to `driverPrototype` to avoid colliding with the reserved JavaScript identifier. +- Normalizes driver `` references (which appear as either bare strings or `{id, ...}` objects in the source XML) into a single object shape with a `.id` field, so downstream code does not need to special-case both forms. +- Output: one JSON file per printer/driver, mirroring the XML structure, under `public/foomatic-db/printer/` and `public/foomatic-db/driver/`. + +### 2. PPD Compilation — `generate-ppds.sh` + +- Invokes `foomatic-compiledb` (from the `foomatic-db-engine` system package) to generate one `.ppd` file per supported printer/driver pair, used for the "Download PPD" / "Preview PPD" features on the printer detail page. +- **Linux/CI only.** On Windows, or when `--skip-ppd` / `SKIP_PPD_GEN=true` is set, this stage is skipped entirely and the pipeline continues without PPDs — `combine-data.ts` simply records `hasPpd: false` for every driver in that case. +- Caches by the foomatic-db git revision: it records the upstream HEAD SHA in `public/ppds/.foomatic-db-revision` and skips regeneration if the SHA hasn't changed and PPDs already exist (unless `--force`/`FORCE_PPD_GEN=true`), avoiding an expensive full recompilation on every CI run when the upstream database hasn't moved. + +### 3. Normalization + Enrichment — `combine-data.ts` + +This is the core data-unification stage. For every printer it: + +- Joins the printer and driver JSON graphs bidirectionally — a printer's `drivers` list and a driver's `printers` list are cross-referenced so each side knows about the other, even when only one side declares the relationship in the source XML. If a driver references a printer ID that has no printer JSON file of its own, a minimal stub printer record is synthesized from the ID (split on `-` into manufacturer/model) rather than silently dropping the driver. +- Derives normalized attributes via pure helper functions (now shared with the test suite in `lib/foomatic/printer-attributes.ts`): + - `getFunctionalityStatus` — maps the raw Foomatic functionality grade (`A`/`B`/`C`/`?`) to a human-readable status (`Perfect`/`Mostly`/`Unknown`/`Unsupported`). + - `getPrinterType` — inspects `` to classify a printer as `inkjet`, `laser`, `dot-matrix`, or `unknown`. + - `getCommandsetTokens` / `normalizeCommandsetToken` — extracts page-description-language tokens from `` (including IEEE1284 `CMD:` device-ID strings) and folds dozens of upstream spelling variants (`PostScript`, `PS2`, `Adobe PostScript`, …) into a small set of canonical tokens (`POSTSCRIPT`, `PCLXL`, `PCL5E`, `PCL`, …). + - `getColorCapability` / `getBooleanCapability` — resolves color support from `` first, falling back to free-text capability fields. + - `getPSLevel` / `getPCLLevel` — parses PostScript/PCL language-level strings into normalized integer tiers. + - `getMaxDpi` — resolves maximum print resolution from ``. +- Resolves each printer's `recommended_driver` (the printer's own declared driver if present, otherwise the first known supporting driver) and builds full driver detail records, including whether a compiled PPD exists for that driver (`hasPpd`/`ppdPath`). +- Output: `public/foomatic-db/printers.json` — the single unified, normalized dataset that every later stage and the frontend both read from. + +### 4. Performance Split — `split-printers.ts` + +`printers.json` contains every printer's full driver/PPD/option detail and is too large to ship to every page. This stage splits it into: + +- `printersMap.json` — a lightweight per-printer projection (`id`, `manufacturer`, `model`, `type`, `status`, `functionality`, `driverCount`, `color`) used by the directory listing page and to drive Next.js's `generateStaticParams()` for static export. +- `printers/.json` — one full-detail file per printer, fetched only when a user visits that printer's detail page. + +### 5. Performance Split — `split-drivers.ts` + +Applies the same treatment to the driver graph, producing `drivers.json`, a lightweight `driversMap.json` index, and one `drivers/.json` per driver for the driver detail pages. + +### 6. Feature Engineering — `vectorize.ts` + +Builds a weighted numeric feature vector for every printer. See [foomatic-recommendation-quality.md](./foomatic-recommendation-quality.md) for the full feature list, weights, and the rationale/timeline behind each one. Output: `cache/foomatic/feature-matrix.json`. + +### 7. Similarity Computation — `compute-similarity.ts` + +For every printer, scores every other printer with the full model described below (IDF-weighted cosine, evidence damping, capability-conflict penalties), keeps the top 10 candidates at or above `MIN_SIMILARITY_SCORE`, orders them deterministically (score, then id — see `lib/foomatic/scoring.ts`), and generates human-readable "why this printer?" explanations. See [foomatic-data-formats.md](./foomatic-data-formats.md) for the exact output schema. + +--- + +## Similarity Methodology Summary + +The recommendation engine is an **engineered similarity model** — weighted feature similarity with explicit, individually-motivated corrections — not a trained/learned model and not an embedding model. This was a deliberate scope choice: the feature space (driver compatibility, page-description-language support, color, resolution tier) is small, well-understood, and directly interpretable, which is what makes the "why this printer?" explanation feature possible. There is no `scikit-learn`-equivalent or embedding library dependency in this pipeline; the only data-processing dependency is `fast-xml-parser` for stage 1. Product-level descriptions may call the portal "AI-driven"; at the implementation level this is deterministic engineered similarity, and the docs use that language. + +The published score for a candidate pair is + +``` +score(a, b) = cos_idf(a, b) * (1 - exp(-k / EVIDENCE_TAU)) * conflictPenalty(a, b) +``` + +computed in this order: + +1. One-hot feature encoding with fixed per-group weights (`vectorize.ts`) +2. IDF scaling of driver-family and command-set dimensions — `idf(t) = ln(1 + N/df(t))`, mean-normalized — so ubiquitous drivers (e.g. `postscript`, 1,746 printers) carry less weight than rare, diagnostic ones +3. Cosine similarity over the weighted vectors +4. Evidence damping: `k` counts dimensions active in **both** vectors; near-empty vectors can no longer score as perfect matches +5. Capability-conflict penalties for known type, colour, and extreme-resolution contradictions (`lib/foomatic/scoring.ts`) +6. `MIN_SIMILARITY_SCORE` floor on the final damped score +7. Top-K selection with deterministic score-then-id ordering +8. Explanation generation (`computeSharedFeatures()`) and per-printer shard output + +Every constant in that chain is documented, with measured justification, in [foomatic-recommendation-quality.md](./foomatic-recommendation-quality.md). + +Computation is brute-force O(n²) — every printer is compared against every other printer. At the current scale of the Foomatic database (several thousand printers), this completes in well under a minute (see `compute-similarity.ts` runtime logging). This is a known, accepted scaling limit; see the **Production Readiness** notes in the project's GSoC midterm audit for the documented ceiling and mitigation options if the dataset grows substantially. + +--- + +## CI/CD Automation + +Two GitHub Actions workflows run the pipeline: + +| Workflow | Trigger | Similarity computation | +|---|---|---| +| `.github/workflows/build.yml` | `pull_request`, `push` to `master`, `workflow_dispatch` | **Runs in full** (nothing is deployed from this workflow) | +| `.github/workflows/deploy.yml` | `push` to `master`, weekly `schedule` (`0 2 * * 1`, Mondays 02:00 UTC), `workflow_dispatch` | **Runs in full** | + +The full pipeline runs automatically as the first half of the `build` script (`yarn generate && next build && ...`), where `yarn generate` invokes `scripts/foomatic/data-generate.ts` before Next.js compiles. Both workflows reach it by running `yarn build`. The weekly cron in `deploy.yml` is what keeps recommendations fresh as the upstream `foomatic-db` repository evolves, without requiring a manual trigger — this satisfies the proposal's "automated GitHub Actions workflows for scheduled data refresh and retraining" deliverable. + +Both workflows install the same system dependencies needed for XML/PPD processing: `foomatic-db-engine`, `foomatic-db-compressed-ppds`, `cups-filters`, `ghostscript`, `bsdmainutils`, `libxml2-utils`, `xsltproc`. + +For the exact commands to reproduce this locally, see [foomatic-retraining.md](./foomatic-retraining.md). diff --git a/docs/foomatic-recommendation-quality.md b/docs/foomatic-recommendation-quality.md new file mode 100644 index 00000000..dc570192 --- /dev/null +++ b/docs/foomatic-recommendation-quality.md @@ -0,0 +1,459 @@ +# Foomatic Recommendation Quality — Evolution and Evidence + +This document quantifies how the similarity engine's recommendation quality changed across this branch's feature-addition commits, and records the measurements behind each change. + +## Methodology + +The pipeline (`scripts/foomatic/vectorize.ts` + `compute-similarity.ts`) is deterministic: same input data + same code = same output. This makes historical comparison straightforward — for each milestone commit below, the **exact pipeline code from that commit** was extracted and re-run against the **current, full `printers.json`** (6,573–6,657 printers, depending on milestone — see table). Holding the dataset fixed and varying only the scoring code isolates the effect of each feature addition from incidental upstream data drift. + +This reproduces the same `logScoreDistribution()`/`logSpotCheck()` diagnostics the pipeline already prints on every run (`compute-similarity.ts`), just captured across history instead of a single point in time. + +| Milestone | Commit | Feature set | Features encoded | +|---|---|---|---| +| Baseline | `a370869` | Driver family + printer type + functionality | 429 | +| +Color | `6a93780` | + color capability | 430 | +| +Commandsets | `f3aab1f` | + PDL token one-hot (PostScript, PCLXL, PCL5E, …) | 455 | +| +PS/PCL level | `73a9161` | + PostScript/PCL capability tier | 459 | +| +Resolution | `2d6f3b7` | + DPI resolution tier | 463 | + +--- + +## Scoring Model + +The published score is not a bare cosine. Three corrections are applied, each +introduced because a measured failure mode demanded it: + +``` +score(a,b) = cos(a,b) . (1 - exp(-k / tau)) . PROD(conflict penalties) +``` + +**1. IDF-weighted features.** Driver-family and command-set dimensions are scaled +by `idf(t) = ln(1 + N / df(t))`, renormalized to mean 1 across the vocabulary. +Sharing `postscript` (1,746 of 6,657 printers) yields weight 0.62; sharing +`necp6` (8 printers) yields 2.64. Without this, every PostScript printer looked +equally similar to every other, and the top-3 was an arbitrary slice of a +1,700-printer cluster. + +**2. Evidence damping.** Cosine on sparse one-hot vectors returns exactly 1.0 +whenever both vectors are near-empty: two printers sharing one dimension, with +every other attribute unknown, are geometrically identical. `k` counts the +dimensions on which both printers are non-zero, and `1 - exp(-k/tau)` with +`tau = 4` maps k = 1 -> 0.22, 4 -> 0.63, 8 -> 0.86, 12 -> 0.95. Before this, +recommendations resting on a single catch-all driver scored *higher* on average +(0.992) than recommendations backed by eleven shared features (0.984). + +**What the score is - and is not.** The score is deterministic and bounded in +[0, 1). It is **not** a probability, not a statistically calibrated confidence, +not a guarantee of printer compatibility, and not a human-validated accuracy +figure. The UI presents it as "N% similarity" under one of four tiers - +**High confidence** (>= 0.85), **Good match** (>= 0.70), **Moderate match** +(>= 0.50), **Limited evidence** (below). The thresholds sit at the midpoints +between the mean scores of the observed shared-feature evidence bands +(<=1 feature: 0.393, 2-3: 0.593, 4-6: 0.787, 7+: 0.921), so each tier reflects +a real difference in supporting evidence rather than a cosmetic slice of the +distribution. Measured on the full artifact set, "High confidence" +recommendations are 90% seven-plus-feature matches with zero capability +conflicts, and all 1,803 generic-driver-only pairs fall in "Limited evidence". + +**3. Capability-conflict penalties.** The vector space can express agreement but +not contradiction, so a mono printer sharing a driver family with a colour one +still scored well despite being unable to substitute for it. Hard substitution +barriers are applied multiplicatively (see the table below). + +--- + +## The Collapse Problem + +The baseline pipeline (driver family + type + functionality only) suffered from a textbook similarity-collapse failure mode: many functionally different printers share the same generic driver, so their feature vectors were nearly identical regardless of actual hardware differences. The measured signature of this is **score saturation** — recommendations clustering at a perfect cosine score of 1.000 even between printers that are not, in fact, equivalent. + +### Score saturation (% of recommendations scoring >= 0.9995, i.e. "exact match") + +Feature engineering alone reduced saturation from 92.0% to 79.8%, but did not +solve it: four in five recommendations were still presented as perfect matches. +The scoring-model corrections above removed it entirely. + +| Stage | Saturated | Mean score | p10 | +|---|---|---|---| +| Baseline (driver + type + functionality) | **92.0%** | 0.996 | 1.000 | +| + colour | 88.1% | 0.972 | 0.967 | +| + commandsets | 84.4% | 0.974 | 0.955 | +| + PS/PCL level | 82.8% | 0.973 | 0.953 | +| + resolution (feature engineering complete) | 79.8% | 0.969 | 0.935 | +| **+ IDF, evidence damping, conflict penalties** | **0.0%** | **0.814** | **0.393** | + +Rows through "feature engineering complete" are measured over all retained top-10 recommendations, the pipeline's historical diagnostic basis. The final row, and every current-state figure in this document, is measured over the 19,606 recommendations actually displayed (top-3 per printer); on that displayed basis, pre-correction saturation was 86.8% (17,212 of 19,827). +The score histogram changed from a single spike (17,212 of 19,827 at 1.0) to a +spread distribution peaking at 0.9 and reaching down to 0.3. + +The decisive metric is the correlation between how much evidence supports a +recommendation and the confidence shown for it: + +| | Before | After | +|---|---|---| +| Corr(shared features, score) | 0.078 | **0.849** | +| Mean score, weak evidence (<= 1 shared feature) | 0.992 | 0.393 | +| Mean score, strong evidence (>= 2) | 0.984 | 0.859 | + +Before, the confidence signal was *inverted*: thin recommendations scored higher +than well-supported ones. It is now strongly aligned. + +### Recommendation churn + +Comparing the **#1 recommendation** for every printer between the baseline and final feature sets: + +> **4,546 of 6,573 printers (69.2%) received a different top recommendation** once color, commandset, PS/PCL level, and resolution features were added. + +This is the direct, measured effect of the four feature-addition commits — not an estimate. A concrete example from that diff: + +- **Alps-MD-2010** (baseline): top recommendation was **Citizen-printiva700** — a different manufacturer entirely, sharing only a generic driver family. +- **Alps-MD-2010** (final): top recommendation became **Alps-MD-2300** — a same-manufacturer, same-series printer, a substantially more sensible suggestion surfaced once commandset and capability features were available to discriminate within the driver-family cluster. + +--- + +## Feature Weights and Tunables + +All weights live as named constants at the top of `scripts/foomatic/vectorize.ts`; thresholds live in `compute-similarity.ts` and `RecommendedPrintersSection.tsx`. They are the knobs to turn when tuning recommendation behaviour. + +| Constant | Value | Where | Effect | +|---|---|---|---| +| `RECOMMENDED_DRIVER_WEIGHT` | 3.0 | `vectorize.ts` | Dominant signal — two printers sharing a preferred Linux driver are strongly similar | +| `COMMANDSET_WEIGHT` | 1.5 | `vectorize.ts` | One-hot per normalized PDL token; highest-weighted post-baseline feature | +| `SUPPORTED_DRIVER_WEIGHT` | 1.0 | `vectorize.ts` | One-hot per additional supported driver family | +| `COLOR_WEIGHT` | 1.0 | `vectorize.ts` | Binary colour capability | +| `LANG_WEIGHT` | 1.0 | `vectorize.ts` | PostScript / PCL support present | +| `RESOLUTION_WEIGHT` | 0.75 | `vectorize.ts` | One-hot across four DPI tiers (≤300, ≤600, ≤1200, >1200) | +| `TYPE_WEIGHT` | 0.5 | `vectorize.ts` | Mechanism class (laser / inkjet / dot-matrix) | +| `LANG_LEVEL_WEIGHT` | 0.5 | `vectorize.ts` | Bonus for matching PostScript 3 / PCL 6 specifically | +| `FUNCTIONALITY_WEIGHT` | 0.25 | `vectorize.ts` | Linux support grade (A/B/C), weakest signal | +| `MIN_COMMANDSET_FREQUENCY` | 20 | `vectorize.ts` | Commandset tokens rarer than this are dropped from the vocabulary as noise | +| `TOP_K` | 10 | `compute-similarity.ts` | Candidates retained per printer | +| `MIN_SIMILARITY_SCORE` | 0.35 | `lib/foomatic/scoring.ts` | Floor on the damped score; removes single-dimension matches automatically | +| `EVIDENCE_TAU` | 4 | `lib/foomatic/scoring.ts` | Damping constant in `1 - exp(-k/tau)`; larger = harsher on thin evidence | +| `TYPE_CONFLICT_PENALTY` | 0.5 | `lib/foomatic/scoring.ts` | Applied when both mechanism types are known and differ | +| `COLOR_CONFLICT_PENALTY` | 0.6 | `lib/foomatic/scoring.ts` | Applied when one prints colour and the other does not | +| `RESOLUTION_CONFLICT_PENALTY` | 0.7 | `lib/foomatic/scoring.ts` | Applied when max resolutions differ by >= 4x | +| `RESOLUTION_CONFLICT_RATIO` | 4 | `lib/foomatic/scoring.ts` | Max-DPI ratio at/above which the resolution penalty applies | +| `CONFIDENCE_HIGH_THRESHOLD` | 0.85 | `lib/foomatic/scoring.ts` | "High confidence" tier floor - midpoint between the 4-6 and 7+ shared-feature evidence-band mean scores | +| `CONFIDENCE_GOOD_THRESHOLD` | 0.7 | `lib/foomatic/scoring.ts` | "Good match" floor - midpoint between the 2-3 and 4-6 evidence bands | +| `CONFIDENCE_MODERATE_THRESHOLD` | 0.5 | `lib/foomatic/scoring.ts` | "Moderate match" floor; below it sits "Limited evidence", where every generic-driver-only pair lands | + +Raising a weight increases how much that attribute pulls two printers together; the vectors are L2-normalized by the cosine denominator, so only the *relative* magnitudes matter. + +--- + +## Feature-by-Feature Timeline + +### Color features (`6a93780`, 2026-06-10) + +**Problem:** Driver-family-only similarity could recommend a monochrome printer in place of a color one, since many mono and color printers in the same product family share a driver. +**Change:** Added a 1.0-weighted binary color feature (`vectorize.ts`). +**Measured effect:** Saturation dropped from 92.0% → 88.1%; mean score dropped from 0.996 → 0.972, the largest single-commit drop in mean score of any feature addition — consistent with color being the single most common source of false-positive matches in the baseline. + +### Commandset features (`f3aab1f`, 2026-06-10) + +**Problem:** Driver family is a coarse compatibility proxy — two printers can share a driver family without supporting the same page-description language (PostScript vs. PCL vs. ESC/P). +**Change:** Added a 1.5-weighted one-hot feature per normalized commandset token (the highest weight of any feature added after the baseline), with `normalizeCommandsetToken()` folding ~20 upstream spelling variants into canonical tokens (`POSTSCRIPT`, `PCLXL`, `PCL5E`, `PCL`, …), filtered to commandsets appearing in at least 20 printers (`MIN_COMMANDSET_FREQUENCY`) to avoid vocabulary noise from rare tokens. +**Measured effect:** Saturation dropped from 88.1% → 84.4%; feature count grew from 430 → 455 (25 new commandset dimensions passed the frequency filter). + +### PostScript/PCL level features (`73a9161`, 2026-06-10) + +**Problem:** Commandset matching alone treats "supports some PostScript" as equivalent regardless of capability tier — a PS1 printer and a full PS3 printer would score identically on the commandset feature. +**Change:** Added separate language-support and level-bonus features (1.0 + 0.5 weight) for PostScript level 3 and PCL level 6, refining within-language-family matching. +**Measured effect:** Saturation dropped from 84.4% → 82.8%. + +### Resolution features (`2d6f3b7`, 2026-06-12) + +**Problem:** Two printers with identical driver, commandset, and language support could still differ enormously in print quality (300 dpi vs. 2400 dpi) and still be scored as near-identical. +**Change:** Added a 0.75-weighted one-hot feature across four DPI tiers (≤300, 300–600, 600–1200, >1200). +**Measured effect:** Saturation dropped from 82.8% → 79.8% — the largest p10 drop of any single addition (0.953 → 0.935), reflecting that resolution tier is a meaningfully independent axis of variation across the dataset. + +### Obsolete driver exclusion (2026-08-13) + +**Problem:** foomatic-db marks some driver entries ``. +The printer and driver pages already badge those as obsolete, but the similarity +features did not filter them, so a dead driver could act as live compatibility +evidence. Measured on the artifacts before this change: **417** of 34,424 +driver-family claims in displayed recommendations named a family that one side +reached *only* through an obsolete driver — including `Shared driver family: +hpdj` as the leading reason on `HP-2000C`, where `hpdj` is obsolete in favour of +`pcl3`. + +**Change:** `getSupportedDriverFamilies()` skips entries with `obsolete === true`, +and `getRecommendedDriverFamily()` resolves an obsolete recommended driver to the +successor foomatic-db names in `replacedBy`. Only the explicitly recorded +replacement is used; none is ever inferred. A replacement is *not* added to the +supported set, because upstream records that a driver is superseded — not that +the successor supports this particular printer. No weight, threshold, or +scoring-formula change accompanies this. + +**Measured effect:** Obsolete-only explanation claims **417 → 0** (now a hard +invariant in `tools/eval/metrics.mjs`). The supported-driver vocabulary shrank +198 → 150 and the feature count 463 → 415, because 48 families existed only via +obsolete entries catalogue-wide. Displayed rankings moved for a small minority: +**157 printers (2.4%)** got a different top recommendation and **271 (4.1%)** a +different top-3, with a mean displayed-score delta of **+0.0007** (range −0.104 +to +0.27). Aggregate quality held: evidence/score correlation 0.849 → 0.851, +type contradictions 0% → 0%, colour 0.20% → 0.21%, ≥4× resolution gaps 0.04% → +0.04%, and the deterministic grading sample's usable share unchanged at 89.7%. + +An obsolete entry that names no successor is handled by the same path: nothing is +substituted, so the printer contributes no preferred-driver evidence at all. A printer +left with only obsolete drivers therefore has no usable driver, and where its recorded +`functionality` grade is unknown its status becomes `"Unsupported"` rather than +remaining unrated. A recorded grade is never overwritten. `metrics.mjs` enforces this +as `printersUnratedWithNoUsableDriver`. + +**Known costs, recorded rather than smoothed over:** + +- `HP-DeskJet_400C` is the one printer that lost all recommendations. Its only + driver entry is the obsolete `gimp-print`, and it has no recorded type, colour, + resolution, or command set, so after the filter there is too little live + evidence to clear `MIN_SIMILARITY_SCORE`. It previously showed three + bottom-tier suggestions whose sole stated reason came from that dead driver; an + empty section is the more honest outcome. +- 39 printers whose top recommendation was a same-manufacturer sibling now show a + different vendor there. **35 of those are exact score ties** where only the + deterministic id tie-break differs, and **none** is a score decrease — 4 score + higher than before. Catalogue-wide the same-manufacturer share of top + recommendations moved 47.57% → 47.00%. +- Individual sibling pairs can still be reordered out of the visible top-3 when + their strongest shared signal was the obsolete driver. `Canon-BJC-620` is the + clearest case: `Canon-BJC-610` moves from first to fourth. It was **not + outranked by better-matching printers** — after the obsolete `bjc610XY.upp` + uniprint profile is excluded, `Canon-BJC-610`, `Canon-BJC-4100` and + `Canon-BJC-4200` have identical live driver sets (`bjc600` + `omni`) and + identical type, colour, resolution and functionality, so all three score + **exactly 0.770**. The three-way tie is resolved by `scoreThenIdComparator`'s + deterministic ascending-id ordering, under which `Canon-BJC-4100` and + `Canon-BJC-4200` sort before `Canon-BJC-610`. What the obsolete profile had + supplied was a seventh shared dimension, lifting `evidenceWeight(7) = 0.826` + over `evidenceWeight(6) = 0.777`; without it the model genuinely cannot + distinguish the three candidates, which is an accurate statement about the live + data even though a maintainer would place the BJC-610 first on generation + grounds. Ties like this are the norm rather than the exception here: 75% of + printers have every retained recommendation tied at one score + (`pctAllRetainedTied`), so this fix pushes additional printers into a + pre-existing weakness rather than creating a new one. +- 44 of the 157 changed top recommendations belong to printers with **no** + obsolete driver at all. Those moved because IDF is renormalized to mean 1 over + the smaller vocabulary, so every supported-driver weight shifts slightly. This + is a side effect of the vocabulary shrinking, not of any per-printer edit. + +### What excluding obsolete drivers costs + +The exclusion is not a strict improvement, and it is worth being precise about +what it gives up. Measured against the pre-filter artifacts: + +| | Families | Median printers covered | Mean | Max | +| --- | --- | --- | --- | --- | +| Dropped (reachable only via obsolete entries) | 48 | 2 | 4.9 | 48 | +| Kept (reachable via a current driver) | 150 | 3 | 79.6 | 3,851 | + +The dropped families are narrow, per-model legacy drivers — `cdj670` (29 +printers), `bjc610xy.upp` (7), `stc600x.upp` (4), `drv_z42` (4). Because IDF +rewards rarity, they were the most discriminative driver evidence available. What +remains are broad umbrellas: `postscript` (3,851), `gutenprint` (1,635), `pdf` +(1,578), `hplip` (472). An obsolete legacy driver was often written for one +model generation, so sharing it was in practice a proxy for **model-generation or +OEM-rebadge kinship** — information the live umbrella drivers cannot express, +because they each cover hundreds of unrelated models. + +**180** printers lost a narrow family. **137** of those retain another narrow +family and are unaffected in practice. **43** lost their only narrow family and +are now ranked purely on umbrella evidence; of those, 21 show a changed top-3 and +3 lost a same-manufacturer top recommendation. That is 0.65% of the catalogue. +The affected set is coherent: mid-range Epson Stylus Color/Photo models, HP +OfficeJet 5xx–7xx and R-series, HP DeskJet 1600C/810C/815C/955C, HP LaserJet +1010/1012, Lexmark Z42/Z43/X73, Compaq-IJ1200, Canon BJC-6000/6200/S450. + +`Compaq-IJ1200` is the clearest loss. Its top three used to be `Lexmark-X73`, +`Lexmark-Z42` and `Lexmark-Z43`, all sharing the obsolete `drv_z42` — and the +Compaq IJ1200 is a rebadged Lexmark, so that driver was effectively an +OEM-identity fingerprint, arguably the most useful kind of match this engine can +surface. Its top three are now unrelated Epson inkjets sharing only +`gutenprint` + inkjet + colour + resolution tier. `Epson-Stylus_Color_II` +similarly lost `Epson-Stylus_Color_IIs`, its own variant, which was linked via +the obsolete `stc2X.upp`. + +None of this argues for treating obsolete drivers as current compatibility +evidence. A recommendation that says two printers are compatible because they +share a driver upstream marks `` is telling the reader +to rely on something foomatic-db says not to use, whatever incidental +hardware-kinship information that entry also carried. The correct way to recover +the signal is a dedicated **model-series / OEM-family feature** derived from live +data — manufacturer plus model-series tokens, and explicit rebadge groupings — +which would restore the Compaq/Lexmark and Stylus Color II/IIs pairings on +defensible grounds. That is a new feature dimension requiring its own +recalibration and is deliberately **out of scope for this change**; it is +recorded here as the intended follow-up. + +Two smaller debits, recorded for completeness: colour-contradiction pairs among +displayed recommendations moved 41 → 42 (3 new, 2 removed), and pairs reaching +"Good match" or above on 4 or fewer shared features moved 10.04% → 10.29%, +because removing dimensions from a target's vector raises its cosine against +every candidate. `Epson-Stylus_Color_1500` crossed 0.588 → 0.712 without any +change to its evidence set. + +--- + +## Driver Count Is Not a Support-Quality Signal + +Raised in review of the recommendation work: the number of drivers supporting a +printer must not be read as a measure of support quality. Many entries can +accumulate for reasons unrelated to how well the printer works — a model that +speaks PostScript and PCL alongside a proprietary language attracts drivers for +each; a PCL printer collects the several near-identical Ghostscript built-ins +plus HPLIP; a proprietary-language printer can attract multiple independent +reverse-engineering efforts of very different maturity; and some entries are +marked obsolete outright. + +The dataset bears this out. Driver-entry count correlates only 0.24 with the +Foomatic `functionality` grade. `Generic-GDI_Printer` carries 9 driver entries at +grade F; `Canon-imageRunner_2800` carries 5 (`lj4dith`, `ljet4`, `ljet4d`, +`Postscript`, `pxlmono`) at grade D; meanwhile 1,558 printers are grade A on a +single driver, and 154 are grade A with no driver entry at all. Of the 1,961 +printers with three or more entries, 44.2% have every family in the generic +PostScript/PCL set. `Kyocera-FS-1000` has 11 entries that reduce to 6 families, +all of them PostScript or PCL5 paths. + +What the pipeline therefore does and does not do: + +- **Driver count is never a scoring input.** There is no count dimension in the + feature vector, nothing in `lib/foomatic/scoring.ts` reads a driver list + length, and no explanation string is derived from one. The recommendation cards + do not display a count either. +- **Shared driver *family* is used as compatibility evidence, and is kept.** + `recommended_driver:` (one-hot, weight 3.0 × IDF) and + `supported_driver:` (set membership, weight 1.0 × IDF) answer whether + two printers can be driven the same way. `normalizeDriverFamily()` folds the + naming variants of one family together, so the four Ghostscript LaserJet + entries above count once, and IDF downweights ubiquitous families so sharing + `postscript` is weak evidence while sharing a rare family is strong. +- **Obsolete drivers are excluded from current compatibility evidence**, per the + timeline entry above. +- **Foomatic exposes no machine-readable measure of driver maturity or + comprehensiveness**, so none is inferred. `obsolete`/`replacedBy` and the + `execution` class are the only quality-adjacent fields available; nothing + distinguishes a comprehensive, actively maintained driver from a minimal + abandoned one. That distinction remains outside what this data supports, and + the pipeline does not pretend otherwise. + +Note what has *not* been solved. Excluding obsolete entries removes dead evidence +and dropping the card badge removes a misleading presentation; neither amounts to +assessing driver quality. Separately, the size of a printer's live driver-family +set still reaches the score indirectly, because more non-zero dimensions raise +the shared-dimension count that `evidenceWeight()` rewards: the correlation +between driver-entry count and mean displayed score is 0.378 (tracked as +`corrDriverCountScore`), essentially unchanged by the obsolete filter. That +coupling is a property of evidence damping — a printer with more recorded +attributes of every kind earns more confidence — and is deliberately left in +place rather than tuned here. + +--- + +## Performance: Per-Printer Data Split (`983f915`, 2026-06-12) + +Not a scoring-quality change, but a necessary scale fix once `recommendations.json` grew large: the full recommendation map for ~6,600 printers serializes to **23.8 MB** at the current snapshot (see `compute-similarity.ts` runtime logging). Shipping that to every printer detail page visit would be a significant and unnecessary page-weight cost, since a single page only ever needs one printer's own top-10 list. + +The fix splits the combined file into one small file per printer (median 3.3 KB, max 6.0 KB at the current snapshot, including denormalized card fields) under `recommendations/.json` (see [foomatic-data-formats.md](./foomatic-data-formats.md#publicfoomatic-dbrecommendationsidjson)), so `RecommendedPrintersSection.tsx` fetches only the relevant slice instead of the full ~24 MB map. + +--- + +## What the Evaluation Proves - and What It Does Not + +The evaluation harness (below) measures internal, mechanically checkable +properties of the generated artifacts. Re-run against the current artifacts it +establishes that: + +- **The engine is discriminative.** 0% of displayed scores saturate at 1.0 + (baseline before the scoring-model corrections: 86.8%). +- **Confidence tracks evidence.** Correlation between shared-feature count and + score is 0.849 (baseline: 0.078, with weak pairs *outscoring* strong ones). + A single-signal pair mathematically cannot leave the bottom tier. +- **Capability conflicts are penalized.** Across displayed recommendations: + type contradictions 0%, colour contradictions 0.2%, >=4x resolution gaps 0.04%. +- **Explanations are truthful.** Every user-visible claim (drivers, command + sets, PS/PCL levels, colour, type, resolution tiers) is re-validated against + `printers.json`: 0 false or misleading claims (baseline: 1,470 false + resolution claims). No claim rests on an obsolete driver + (`claimsCitingObsoleteOnlyFamily` = 0, down from 417). Both are enforced as + hard invariants: `metrics.mjs` exits non-zero if either regresses. +- **Output is reproducible.** Deterministic tie-breaking; identical inputs + produce byte-identical artifacts. +- **Under the documented deterministic rubric** (see `tools/eval/grade.mjs`), + 89.7% of the 195-recommendation stratified sample grades Excellent or Good + ("qualitatively usable") and 4.1% grades Weak or Incorrect. + +It does **not** establish human-validated relevance. There is no labelled +ground-truth dataset in this repository asserting "printer A is a good +alternative to printer B", so figures like the 89.7% above are rubric outcomes +on a deterministic sample - they must not be read as "89.7% accurate" or as +compatibility accuracy. Building a small hand-curated pair set is the single +highest-value follow-up if recommendation quality needs stronger defence. + +--- + +## Known Limitations + +- **Identical feature vectors.** Only 1,096 distinct engineered feature vectors + exist across 6,657 printers; 90.7% of printers share their vector with at + least one other, and in 85.3% of top-3 lists the candidates are byte-identical + clones of each other. Given the recorded Foomatic data these printers + *genuinely cannot be distinguished*; deterministic id ordering is used so tied + results are reproducible, not to pretend to a ranking the data cannot support. +- **Zero-recommendation printers.** 71 printers (1.07%) currently receive no + recommendations: every candidate falls below `MIN_SIMILARITY_SCORE`, almost + always because the printer record carries too little data to accumulate + evidence. The UI shows an explicit empty state for these. +- **Catalogue coverage.** Roughly 27% of printers ever appear in someone's + top-3; the rest are never surfaced. Driver-family clustering concentrates + exposure on a minority of the catalogue. +- **Same-manufacturer and rebadge concentration.** 47.5% of displayed + recommendations are same-manufacturer, and 58.7% of cross-manufacturer ones + pair printers within the Ricoh badge family (Ricoh/Lanier/NRG/Gestetner/ + Savin/Infotec) - technically correct for compatibility, since these are + frequently the same hardware, but of limited value as *alternatives*. Roughly + 22% of recommendations cross to a genuinely different vendor. +- **No human-labelled ground truth** - the most important limitation; see the + section above. +- **O(n^2) computation.** Every printer is compared against every other + (~44M pairs at 6,657 printers), measured at roughly 60-70s single-threaded + on a typical development machine. Acceptable at the current database size and + re-run only at build/regeneration time, but not unbounded. + +--- + +## Reproducing the Evaluation + +Prerequisites: `yarn install`, then generated artifacts in +`public/foomatic-db/` and `cache/foomatic/` (`yarn generate`, or `yarn foomatic:pipeline` for the +foomatic stages alone - see [foomatic-retraining.md](./foomatic-retraining.md)). + +``` +yarn foomatic:eval +``` + +runs three deterministic checks against the artifacts on disk: + +1. **`tools/eval/metrics.mjs`** - full-dataset ranking metrics: score + distribution and saturation, tie rates, evidence/score correlation, + explanation-truthfulness validation of every displayed claim, + manufacturer/rebadge/coverage diversity, and capability contradictions. +2. **`tools/eval/check-docs.mjs`** - verifies every tunable documented in this + file against the values in the source, and greps the docs for retired + terminology. Fails non-zero on drift. +3. **`tools/eval/grade.mjs`** - applies the fixed rubric documented in its + header to a deterministic stratified sample (~68 printers across technology, + colour, resolution, rarity, and manufacturer strata built by + `tools/eval/sample.mjs`; sorted pools, evenly-spaced picks, no randomness - + the same artifacts always yield the same sample, so results are not + cherry-picked). Pass `--rows` for the per-pair detail. + +`node tools/eval/pairs.mjs` prints a maintainer-facing deep-dive of the sampled +pairs - full source/target capabilities beside the user-visible explanation, +bucketed into good / low-value / weak / problematic. + +The harness proves internal consistency, truthfulness, and discriminative +behaviour. It does not prove human-validated relevance (no labelled pair set +exists), and the rubric in `grade.mjs` is itself an engineered heuristic - a +lens for review, not ground truth. diff --git a/docs/foomatic-retraining.md b/docs/foomatic-retraining.md new file mode 100644 index 00000000..d4b8dd17 --- /dev/null +++ b/docs/foomatic-retraining.md @@ -0,0 +1,111 @@ +# Foomatic Pipeline — Regeneration Guide + +This is the operational reference for re-running the printer recommendation pipeline described in [foomatic-pipeline-architecture.md](./foomatic-pipeline-architecture.md). There is no "training" in the machine-learning-model sense — the pipeline deterministically regenerates static artifacts from upstream `foomatic-db` XML and engineered feature weights. "Retraining" here means re-running the pipeline end-to-end after either the upstream database or the feature/weight logic has changed. + +--- + +## Prerequisites + +The full pipeline (including PPD compilation) requires the same system packages installed in CI (`.github/workflows/build.yml` / `deploy.yml`): + +``` +foomatic-db-engine foomatic-db-compressed-ppds cups-filters ghostscript bsdmainutils libxml2-utils xsltproc +``` + +On Windows (or any environment without `foomatic-compiledb`), PPD compilation is automatically skipped — `data-generate.ts` skips it whenever `process.platform === "win32"`, and `generate-ppds.sh` itself exits early if the `foomatic-compiledb` binary isn't found. The rest of the pipeline runs identically either way; printers simply get `hasPpd: false`. + +Node dependencies are managed with `yarn` (`yarn.lock` is the lockfile of record). + +--- + +## Running the full pipeline + +```bash +yarn foomatic:pipeline +``` + +This runs `scripts/foomatic/data-generate.ts`, which executes every stage in order and stops immediately if any stage fails: + +1. `generate-from-xml.ts` — clones/pulls `OpenPrinting/foomatic-db` into `cache/foomatic-db`, converts XML to JSON +2. `generate-ppds.sh` — compiles PPDs (skipped on Windows or with `--skip-ppd`) +3. `combine-data.ts` — produces `printers.json` +4. `split-printers.ts` — produces `printersMap.json` and `printers/.json` +5. `vectorize.ts` — produces `cache/foomatic/feature-matrix.json` (skipped with `--skip-similarity`) +6. `compute-similarity.ts` — produces `cache/foomatic/recommendations.json` and `public/foomatic-db/recommendations/.json` (skipped with `--skip-similarity`) + +This also runs automatically before every `next build`: the `build` script is `yarn generate && next build && ...`, and `yarn generate` invokes this pipeline first. + +### Flags / environment variables + +| Flag | Env var equivalent | Effect | +|---|---|---| +| `--skip-ppd` | n/a (Windows always skips) | Skip PPD compilation (stage 2) | +| `--skip-similarity` | `FOOMATIC_SKIP_SIMILARITY=1` | Skip stages 6–7 (vectorize + compute-similarity) | +| `--force` (forwarded to `generate-ppds.sh` only) | `FORCE_PPD_GEN=true` | Force PPD recompilation even if the cached `foomatic-db` git revision hasn't changed | + +Example — fast local iteration on UI changes without recomputing recommendations: + +```bash +yarn foomatic:pipeline --skip-similarity +``` + +--- + +## Running individual stages + +Each stage can be run independently via its own `package.json` script — useful when iterating on one stage without re-running the whole pipeline: + +```bash +yarn foomatic:generate:xml # stage 1: XML → JSON +yarn foomatic:generate:ppds # stage 2: PPD compilation (Linux/macOS only) +yarn foomatic:data:combine # stage 3: normalization → printers.json +yarn foomatic:data:split # stages 4-5: printersMap.json + printers/.json, + # driversMap.json + drivers/.json +yarn foomatic:data:vectorize # stage 6: cache/foomatic/feature-matrix.json +yarn foomatic:data:similarity # stage 7: cache/foomatic/recommendations.json + recommendations/.json +``` + +Stages 3–7 each read their input from a fixed path — under `public/foomatic-db/`, or `cache/foomatic/` for the feature matrix (see [foomatic-data-formats.md](./foomatic-data-formats.md)) rather than from in-memory state, so they can be re-run independently as long as the upstream artifact they depend on already exists. Running `yarn foomatic:data:vectorize` without first running `yarn foomatic:data:combine` will fail fast with a message telling you which earlier command to run. + +--- + +## When to re-run which stage + +| Change | Stages to re-run | +|---|---| +| Upstream `foomatic-db` content changed (new printers/drivers) | All stages (`yarn foomatic:pipeline`) | +| Editing normalization logic in `lib/foomatic/printer-attributes.ts` (e.g. a new commandset token mapping) | `combine-data` onward (3–6) | +| Editing feature weights or adding a new feature in `vectorize.ts` | `vectorize` onward (5–6) — `combine-data`/`split-printers` output is unaffected | +| Editing similarity scoring/explanation logic in `compute-similarity.ts` | `compute-similarity` only (6) | +| UI-only changes (components, pages) | No pipeline re-run needed — just `next dev` / `next build` against existing `public/foomatic-db/` artifacts | + +--- + +## CI automation + +You do not need to run the pipeline manually for production — it runs automatically: + +- **Every PR / push to `master`:** `build.yml` runs the full pipeline, including similarity computation, so a PR that breaks any stage fails its checks before merge. Nothing is deployed from this workflow. +- **Every push to `master`, plus a weekly cron (`0 2 * * 1`, Mondays 02:00 UTC):** `deploy.yml` runs the full pipeline including similarity computation, then deploys the static export to GitHub Pages. This is what keeps recommendations fresh as `foomatic-db` evolves upstream, without anyone needing to remember to trigger it. + +To manually trigger a full regeneration in CI without waiting for the weekly cron, use the `workflow_dispatch` trigger on `deploy.yml` from the Actions tab. + +--- + +## Verifying a regeneration was a no-op (regression check) + +If you've changed code but expect output to be unchanged (e.g. a refactor with no behavior change), diff the artifacts before and after: + +```bash +cp public/foomatic-db/printers.json /tmp/printers.before.json +yarn foomatic:data:combine +diff -q /tmp/printers.before.json public/foomatic-db/printers.json +``` + +Repeat the same pattern for `cache/foomatic/feature-matrix.json` (after `yarn foomatic:data:vectorize`) and `cache/foomatic/recommendations.json` (after `yarn foomatic:data:similarity`). This is how the `lib/foomatic/*` extraction refactor (see the test suite under `lib/foomatic/__tests__/`) was verified to be behavior-preserving before being merged. + +--- + +## Adding a new similarity feature + +See [foomatic-ui-extending.md](./foomatic-ui-extending.md#adding-a-new-recommendation-signal) for the step-by-step process — it touches `combine-data.ts` (derive the attribute), `vectorize.ts` (encode it with a weight), and `compute-similarity.ts` (optionally add it to `computeSharedFeatures()` for the "why this printer?" explanation). diff --git a/docs/foomatic-ui-extending.md b/docs/foomatic-ui-extending.md new file mode 100644 index 00000000..db6ed52e --- /dev/null +++ b/docs/foomatic-ui-extending.md @@ -0,0 +1,124 @@ +# Foomatic Web Interface — Extending Guide + +This document is for contributors working on the printer discovery and recommendation interface at `/foomatic`. It assumes you've read [foomatic-pipeline-architecture.md](./foomatic-pipeline-architecture.md) and [foomatic-data-formats.md](./foomatic-data-formats.md) for how the data this UI consumes is produced. + +This is the GSoC "Track B" deliverable: *a client-side-only interface, with no backend, consuming the static ML artifacts generated by Track A.* + +--- + +## Component Structure + +``` +app/foomatic/ + printers/page.tsx Directory/listing page (client component) + printer/[make]/[id]/page.tsx Printer detail route — server component, + generateStaticParams() pre-renders one + page per printer from printersMap.json + drivers/page.tsx Driver directory listing + driver/[id]/page.tsx Driver detail route + ppd-o-matic/page.tsx PPD preview route + +components/foomatic/ + Printers.tsx Grid wrapper, renders PrinterCard per item + PrinterCard.tsx Single printer summary card (directory page) + PrinterSearch.tsx Search input + filter controls + PrinterPageClient.tsx Printer detail page body (client component) + RecommendedPrintersSection.tsx "Similar printers" section on the detail page + DriverListClient.tsx Driver directory body + DriverPageClient.tsx Driver detail page body + PpdViewerClient.tsx PPD text viewer (ppd-o-matic route) + shared.tsx Shared primitives: FoomaticCard, FoomaticBadge, + FoomaticStatusBadge, FoomaticSelect, etc. + +lib/foomatic/ + types.ts Printer / Driver / PrinterSummary types + utils.ts calculateAccurateStatus() + sanitize.ts sanitizeFoomaticHtml() — DOMPurify wrapper + base-path.ts withBasePath() — GitHub Pages basePath helper + driver-family.ts normalizeDriverFamily() and friends (also used + by the pipeline scripts — see below) + printer-attributes.ts Pipeline-side attribute derivation (also used + by the pipeline scripts) + similarity-math.ts cosineSimilarity() and friends (pipeline-side) + hooks/use-debounce.ts useDebounce() — used by PrinterSearch +``` + +Everything under `app/foomatic` and `components/foomatic` is a Next.js **client component** (`"use client"`) except `app/foomatic/printer/[make]/[id]/page.tsx`, which is a server component whose only job is to call `generateStaticParams()` (reading `printersMap.json` from disk at build time) and hand off to `PrinterPageClient`. This split exists because static export needs `generateStaticParams()` to run at build time in Node, while the actual data-fetching and interactivity happens in the browser against the static JSON artifacts. + +`lib/foomatic/driver-family.ts`, `printer-attributes.ts`, and `similarity-math.ts` are shared between the **build-time pipeline** (`scripts/foomatic/*.ts`) and its **test suite** (`lib/foomatic/__tests__/`) — they are not UI code, but they live in `lib/foomatic` because that's the one place both the pipeline scripts and (potentially) the frontend can import from without a circular path back into `scripts/`. + +--- + +## Printer Page Architecture + +`app/foomatic/printer/[make]/[id]/page.tsx` → `PrinterPageClient.tsx`: + +1. On mount, fetches `public/foomatic-db/printers/${printerId}.json` (the per-printer split file — see data formats doc). +2. Renders a loading skeleton while the fetch is in flight, and a "Printer not found" card if the fetch fails or 404s — never a hard error. +3. Recomputes `status` via `calculateAccurateStatus()` rather than trusting the printer's raw `status` field directly, because `status` in the data can be stale relative to `functionality`/`driverCount` depending on how the record was synthesized upstream (see `lib/foomatic/utils.ts`). +4. Renders a grouped **Capabilities** block (color, duplex, max resolution, connectivity, page description languages) and the driver list, sorted so `recommended_driver` appears first. PostScript/PCL levels are surfaced through the page-description-language labels rather than as separate rows, since the `commandsets` labels already carry the level (`PCL 5e`, `PostScript 3`). +5. Renders `driver.comments` and `printer.notes` through `sanitizeFoomaticHtml()` before `dangerouslySetInnerHTML` — **never bypass this**; that content originates from upstream contributor-editable XML (see [foomatic-data-formats.md](./foomatic-data-formats.md#publicfoomatic-dbprintersjson)). +6. Renders `` at the bottom. + +--- + +## Recommendation UI + +`RecommendedPrintersSection.tsx`: + +1. Fetches exactly one file: `public/foomatic-db/recommendations/${printerId}.json`. Each entry already carries the `manufacturer`/`model`/`status`/`type` fields the card renders, denormalized by `compute-similarity.ts`, so no second lookup against `printersMap.json` is needed. A driver count is deliberately not among them — do not add one back (see [foomatic-recommendation-quality.md](./foomatic-recommendation-quality.md#driver-count-is-not-a-support-quality-signal)). +2. If you add a field to the card, add it to the denormalization block in `compute-similarity.ts` (and to the schema in [foomatic-data-formats.md](./foomatic-data-formats.md#publicfoomatic-dbrecommendationsidjson)) rather than reintroducing a `printersMap.json` fetch here — that file is ~1.5 MB and would dominate the page's transfer cost. +3. Renders only the **top 3** recommendations (`recs.slice(0, 3)`) — the underlying data has up to 10, the UI deliberately shows fewer to keep the section compact. +4. `ConfidenceBadge` maps the score to one of four evidence-aligned tiers via `confidenceTier()` in `lib/foomatic/scoring.ts`: **High confidence** (≥ 0.85, emerald), **Good match** (≥ 0.70, sky), **Moderate match** (≥ 0.50, amber), **Limited evidence** (below, muted), with a secondary `N% similarity` line. The wording is deliberate: the score is an engineered similarity value, not a probability that the printer will work, so nothing is ever labelled "exact". +5. `sharedFeatures` (the explanation strings produced by `computeSharedFeatures()` in the pipeline) are rendered as a labeled chip list under "Why this printer?". +6. Uses a `cancelled` flag in the `useEffect` cleanup to avoid calling `setState` after unmount (relevant if a user navigates away mid-fetch). +7. `SimilarPrintersTeaser` (exported from the same file) renders a compact card in the printer-page header beside the Recommended driver box: the top similar printer, its `N% similarity` and confidence tier, and an anchor link to `#similar-printers` on the full section. It shares one shard request with the full section through a module-level per-printer cache, and renders nothing while loading or when the printer has no recommendations. +8. Inside recommendation cards the candidate's own Foomatic grade badge is prefixed with "Linux support:" specifically to distinguish the candidate's Linux support grade from the similarity result; the bare badge used elsewhere on the site is unchanged. + +--- + +## Filter System + +`app/foomatic/printers/page.tsx` owns all filter/search/pagination state for the directory listing: + +- State is persisted to `localStorage` under the `STORAGE_KEYS` constants (search query, manufacturer, driver type, mechanism type, support level, color capability, current page, items-per-page) — so filters survive a reload or back-navigation. +- `PrinterSearch.tsx` is a presentational component: it owns the debounced text input (`useDebounce`, 300ms) and renders the filter `