Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .agents/GOALS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,10 @@ This document captures the high-level goals for @libpdf/core. Use this to steer

- [x] **Extract text** — Get text content from pages (extractText API)
- [x] **Search text** — Find text patterns with bounding box results (findText API)
- [x] **Preserve reading order** — Line grouping based on baseline/font
- [x] **Preserve reading order** — Run-based line grouping (strings stay intact, overlaps split)
- [x] **Extract from form XObjects** — Headers, footers, embedded pages, flattened fields
- [ ] **Extract from annotations** — Include comment text, form values
- [ ] **Respect clipping** — Drop text outside the current clip path

### 9. Creation

Expand Down
43 changes: 34 additions & 9 deletions .agents/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,20 +248,45 @@ Embedded file specification handling.

Text extraction from PDF content streams with position tracking.

| Component | Purpose |
| --------------- | ------------------------------------------------ |
| `TextExtractor` | Parses content streams, tracks text state |
| `TextState` | Manages text matrix, font, and positioning |
| `LineGrouper` | Groups characters into lines based on baseline |
| `text-search` | String and regex search with bounding boxes |
| `types.ts` | TextChar, TextLine, TextSpan, SearchResult types |
| Component | Purpose |
| --------------- | --------------------------------------------------------------------- |
| `TextExtractor` | Interprets content streams and form XObjects, emits positioned glyphs |
| `TextResources` | Fonts and form XObjects reachable from a `/Resources` dict, cached |
| `TextState` | Graphics/text state; glyph geometry through Tm × CTM |
| `LineGrouper` | Groups glyphs into runs, runs into lines, lines into spans |
| `text-search` | String and regex search with bounding boxes |
| `types.ts` | ExtractedChar, TextLine, TextSpan, TextMatch types |

### Supported Text Operators

- Positioning: `Td`, `TD`, `Tm`, `T*`
- Showing: `Tj`, `TJ`, `'`, `"`
- Showing: `Tj`, `TJ`, `'`, `"` (codes split via the font's CMap; `Tw` on single-byte code 32)
- State: `Tf`, `Tc`, `Tw`, `Tz`, `TL`, `Ts`, `Tr`
- Graphics: `cm`, `q`, `Q` (matrix transformations)
- Graphics: `cm`, `q`, `Q` (font and text state are part of the saved state)
- XObjects: `Do` for `/Subtype /Form` (own or inherited resources, `/Matrix`, cycle and depth guard)

### Line Grouping Model

Glyphs arrive in stream order. Consecutive glyphs that advance along one baseline form a
**run** (the string a producer drew as a unit). Runs are placed on lines: a run joins a
line if it shares the baseline and none of its space-occupying glyphs collide with glyphs
already there; otherwise it opens a new line on the same baseline. This keeps strings drawn
over each other (tags over invisible labels, layered text) intact instead of interleaving
them glyph by glyph. Within a line, runs are k-way merged by glyph x while intra-run order
is preserved.

Rules layered on that model, each with a fixture or unit test:

- Whitespace and overlay glyphs (combining marks, accents) do not occupy space.
- A glyph repeating one already on the line at the same position is dropped (fake bold,
redrawn chunk boundaries).
- Whitespace-only runs stay with the run drawn before them.
- Lines whose glyphs predominantly move left in stream order keep stream order (design-tool
exports, RTL).

Known limitations: rotated/vertical text, multi-column pages sharing a baseline, an
invisible text layer that duplicates visible text at _different_ positions (yields a
duplicate line), text clipped away by `W n`, and `Tr 3` text is extracted like any other.

## Annotations Layer (`src/annotations/`)

Expand Down
Binary file added fixtures/text/overlapping-invisible-labels.pdf
Binary file not shown.
77 changes: 10 additions & 67 deletions src/api/pdf-page.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,12 @@ import type {
} from "#src/drawing/types";
import { resolveRotationOrigin } from "#src/drawing/types";
import { EmbeddedFont } from "#src/fonts/embedded-font";
import { parseFont } from "#src/fonts/font-factory";
import type { PdfFont } from "#src/fonts/pdf-font";
import {
getEncodingForStandard14,
getStandard14BasicMetrics,
isStandard14Font,
isWinAnsiStandard14,
} from "#src/fonts/standard-14";
import { parseToUnicode } from "#src/fonts/to-unicode";
// Annotation utilities - imported here to avoid dynamic require issues
import { concatBytes } from "#src/helpers/buffer";
import { black } from "#src/helpers/colors";
Expand Down Expand Up @@ -127,6 +124,7 @@ import { PdfStream } from "#src/objects/pdf-stream";
import { PdfString } from "#src/objects/pdf-string";
import { getPlainText, groupCharsIntoLines } from "#src/text/line-grouper";
import { TextExtractor } from "#src/text/text-extractor";
import { TextResources } from "#src/text/text-resources";
import { searchPage } from "#src/text/text-search";
import type { ExtractTextOptions, FindTextOptions, PageText, TextMatch } from "#src/text/types";

Expand Down Expand Up @@ -2817,12 +2815,12 @@ export class PDFPage {
// Get content stream bytes
const contentBytes = this.getContentBytes();

// Create font resolver
const resolveFont = this.createFontResolver();

// Extract characters
const extractor = new TextExtractor({ resolveFont });
const chars = extractor.extract(contentBytes);
const resources = new TextResources(
this.resolveInheritedResources(),
this.ctx.resolve.bind(this.ctx),
);
const chars = new TextExtractor(resources).extract(contentBytes);

// Group into lines and spans
const lines = groupCharsIntoLines(chars);
Expand Down Expand Up @@ -2936,8 +2934,11 @@ export class PDFPage {
private resolveInheritedResources(): PdfDict | null {
// Start with the page dict
let currentDict: PdfDict | null = this.dict;
const visited = new Set<PdfDict>();

while (currentDict && !visited.has(currentDict)) {
visited.add(currentDict);

while (currentDict) {
// Check for Resources on the current node
const resources = currentDict.get("Resources", this.ctx.resolve.bind(this.ctx));

Expand All @@ -2957,62 +2958,4 @@ export class PDFPage {

return null;
}

/**
* Create a font resolver function for text extraction.
*/
private createFontResolver(): (name: string) => PdfFont | null {
// Get the page's Font resources (may be a ref or inherited from parent)
const resourcesDict = this.resolveInheritedResources();

if (!resourcesDict) {
return () => null;
}

const font = resourcesDict.getDict("Font", this.ctx.resolve.bind(this.ctx));

if (!font) {
return () => null;
}

// Preload all font dictionaries and build the cache
const fontCache = new Map<string, PdfFont>();

for (const [key, entry] of font) {
const name = key.value;
const resolved = entry instanceof PdfRef ? this.ctx.resolve(entry) : entry;

let entryDict = resolved instanceof PdfDict ? resolved : null;

if (!entryDict) {
continue;
}

// Parse ToUnicode CMap if present
let toUnicodeMap = null;

const toUnicode = entryDict.get("ToUnicode", this.ctx.resolve.bind(this.ctx));
const toUnicodeStream = toUnicode instanceof PdfStream ? toUnicode : null;

if (toUnicodeStream) {
try {
toUnicodeMap = parseToUnicode(toUnicodeStream.getDecodedData());
} catch {
// ToUnicode parsing failed - continue without it
}
}

// Parse the font
const pdfFont = parseFont(entryDict, {
resolver: this.ctx.resolve.bind(this.ctx),
toUnicodeMap,
});

fontCache.set(name, pdfFont);
}

return (name: string): PdfFont | null => {
return fontCache.get(name) ?? null;
};
}
}
52 changes: 52 additions & 0 deletions src/content/parsing/content-tokenizer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,58 @@ describe("ContentTokenizer", () => {
expect(tokenizer.nextToken()).toEqual({ type: "number", value: 42 });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "m" });
});

it("ends an array at an operator and re-emits the operator", () => {
const tokenizer = new ContentTokenizer(encode("[(a) 5 TJ (b) Tj"));

expect(tokenizer.nextToken()).toEqual({
type: "array",
items: [
{ type: "string", value: encode("a"), hex: false },
{ type: "number", value: 5 },
],
});
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "TJ" });
expect(tokenizer.nextToken()).toMatchObject({ type: "string" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "Tj" });
});

it("keeps the re-emitted operator in order when peeked", () => {
const tokenizer = new ContentTokenizer(encode("[(a) TJ q"));

expect(tokenizer.peek()).toMatchObject({ type: "array" });
expect(tokenizer.nextToken()).toMatchObject({ type: "array" });
expect(tokenizer.peek()).toEqual({ type: "operator", name: "TJ" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "TJ" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "q" });
});

it("skips a dict entry whose key is not a name", () => {
const tokenizer = new ContentTokenizer(encode("<< 1 2 /K 3 >> BDC"));

expect(tokenizer.nextToken()).toEqual({
type: "dict",
entries: new Map([["K", { type: "number", value: 3 }]]),
});
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "BDC" });
});

it("ends a dict at an operator and re-emits the operator", () => {
const tokenizer = new ContentTokenizer(encode("<< /K 1 /V BDC q"));

expect(tokenizer.nextToken()).toMatchObject({ type: "dict" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "BDC" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "q" });
});

it("stray ) or } does not stall the tokenizer", () => {
const tokenizer = new ContentTokenizer(encode("q ) } Q"));

expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "q" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: ")" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "}" });
expect(tokenizer.nextToken()).toEqual({ type: "operator", name: "Q" });
});
});

describe("peek and eof", () => {
Expand Down
26 changes: 22 additions & 4 deletions src/content/parsing/content-tokenizer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ export class ContentTokenizer {

private peeked: ContentOrOperatorToken | null = null;

/** An operator found inside an unterminated array or dict, emitted next */
private pushedBack: OperatorToken | null = null;

constructor(bytes: Uint8Array) {
this.scanner = new Scanner(bytes);
this.reader = new TokenReader(this.scanner);
Expand Down Expand Up @@ -90,6 +93,13 @@ export class ContentTokenizer {
}

private readNext(): ContentOrOperatorToken | null {
if (this.pushedBack !== null) {
const token = this.pushedBack;
this.pushedBack = null;

return token;
}

while (true) {
const token = this.reader.nextToken();

Expand Down Expand Up @@ -188,8 +198,9 @@ export class ContentTokenizer {
}

if (item.type === "operator") {
// Operators shouldn't appear inside arrays - treat as error
throw new Error(`Unexpected operator in array: ${item.name}`);
// The closing ] was lost; end the array and let the operator run
this.pushedBack = item;
break;
}

items.push(item);
Expand Down Expand Up @@ -220,8 +231,14 @@ export class ContentTokenizer {
break;
}

if (key.type === "operator") {
// The closing >> was lost; end the dict and let the operator run
this.pushedBack = key;
break;
}

if (key.type !== "name") {
throw new Error(`Expected name as dict key, got ${key.type}`);
continue;
}

// Read value
Expand All @@ -232,7 +249,8 @@ export class ContentTokenizer {
}

if (value.type === "operator") {
throw new Error(`Unexpected operator as dict value: ${value.name}`);
this.pushedBack = value;
break;
}

entries.set(key.value, value);
Expand Down
2 changes: 1 addition & 1 deletion src/document/forms/form-font.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ describe("ExistingFont", () => {
baseFontName: "StubCID",
cidToGidMap: new Uint16Array([0, 7]),
embeddedProgram: new StubFontProgram(new Map([[0x0041, 7]]), new Set([7])),
toUnicodeMap: new ToUnicodeMap(new Map([[1, "A"]])),
toUnicodeMap: ToUnicodeMap.fromEntries([[1, "A"]]),
});
const font = new ExistingFont("F0", null, null, true, cidFont);

Expand Down
21 changes: 21 additions & 0 deletions src/fontbox/cmap/parser.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -322,4 +322,25 @@ describe("CMap Parser", () => {
expect(cmap.wmode).toBe(0);
});
});

describe("malformed dictionaries", () => {
it("recovers from an unterminated << (PDFBox readDictionary semantics)", () => {
// From genko_oc_shiryo1.pdf: the CIDSystemInfo dict is never closed.
const cmap = parseCMap(
new TextEncoder().encode(`/CIDInit /ProcSet findresource begin 12 dict begin begincmap
CIDSystemInfo <</Registry (F1+0) /Ordering (F1) /Supplement 0
/CMapName /F1+0 def
/CMapType 2 def
1 begincodespacerange <0002> <3A2A> endcodespacerange
2 beginbfchar
<0002> <0020>
<0519> <2461>
endbfchar
endcmap`),
);

expect(cmap.toUnicodeBytes(new Uint8Array([0x05, 0x19]))).toBe("\u2461");
expect(cmap.toUnicodeBytes(new Uint8Array([0x00, 0x02]))).toBe(" ");
});
});
});
10 changes: 4 additions & 6 deletions src/fontbox/cmap/parser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -510,12 +510,10 @@ class CMapParser {
const dict = new Map<string, unknown>();
let key = this.parseNextToken();

while (key !== null && !(isOperator(key) && key.op === ">>")) {
if (isName(key)) {
const value = this.parseNextToken();
dict.set(key.name, value);
}

// As in PDFBox, stop at the first non-name key so an unterminated
// dictionary doesn't swallow the rest of the CMap.
while (isName(key)) {
dict.set(key.name, this.parseNextToken());
key = this.parseNextToken();
}

Expand Down
2 changes: 1 addition & 1 deletion src/fonts/cid-font.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ describe("CIDFont", () => {
subtype: "CIDFontType2",
baseFontName: "TestFont",
cidToGidMap: new Uint16Array([0, 2]),
toUnicodeMap: new ToUnicodeMap(new Map([[1, "A"]])),
toUnicodeMap: ToUnicodeMap.fromEntries([[1, "A"]]),
});

expect(font.tryGetCharCodeForUnicode(0x41)).toBe(1);
Expand Down
Loading
Loading