Skip to content

Refuse JSON nested deeper than maxdepth instead of overflowing the stack - #498

Open
quinnj wants to merge 2 commits into
masterfrom
jq/parse-maxdepth
Open

quinnj wants to merge 2 commits into
masterfrom
jq/parse-maxdepth

Conversation

@quinnj

@quinnj quinnj commented Oct 5, 2026

Copy link
Copy Markdown
Member

Problem

Parsing recurses once per nested object or array. Input such as repeat("[", 100_000) (100 KB, well under typical request-size limits) throws StackOverflowError ("program state may be corrupted") from every entry point: untyped parse, typed parse(json, T), lazy(...)[], parse!, isvalidjson, and skipping unknown members. A server parsing untrusted bodies has no clean way to reject it.

Change

Every parse path enters a container through applyobject, applyarray, or the tuple maketuple. Each now checks a nesting depth carried on the LazyValue and, before recursing, throws:

ArgumentError: JSON nesting depth at byte position 513 exceeds the `maxdepth` limit of 512; pass a larger `maxdepth` keyword argument to parse deeper input
  • New maxdepth keyword on lazy/parse/parse!/isvalidjson (default 512), passed like allownan or jsonlines.
  • On master, the shallowest path (a recursive struct) overflowed at about 2,000 levels, so 512 keeps about 4x headroom.
  • With error_context=true, the error arrives as the ParseError cause, like other errors.

Performance

  • The depth and limit are Int32 fields in existing struct padding. LazyValue (80 bytes), LazyOptions (48), and LazyObject/LazyArray (72) keep their sizes.
  • Allocations are byte-for-byte unchanged on all 56 read cases measured (the benchmarks/ cases plus larger realistic and container-only stress inputs).
  • Instructions retired per call, measured against master: +0.4% to +2% on ordinary documents, and up to +5–6% on inputs made only of empty containers. The check is 5 instructions per object or array entered.
  • Cycle counts and wall-clock time showed no difference outside measurement noise.
  • The same 42 methods compile at first call, and the package image grows 0.3%.

Tests

  • New nesting deeper than maxdepth testset. It covers 14 entry points/targets, the 512/513 boundary, raising and lowering maxdepth, and error_context.
  • The trim entrypoint workload now compiles the limit error path and the maxdepth keyword under --trim.
  • Two struct-target cases added to the existing truncated-input testset.

🤖 Generated with Claude Code

…stack

Parsing recurses once per nested object or array, so input such as
`repeat("[", 100_000)` (100 KB, far below typical request-size limits)
threw StackOverflowError — "program state may be corrupted" — from every
entry point: untyped, typed (StructUtils make), lazy materialization,
parse!, isvalidjson and skipping of unknown members. A server that parses
untrusted bodies had no clean way to reject it.

Every parse path enters a container through `applyobject`, `applyarray`
or the tuple `maketuple`, so each now checks a nesting depth carried on the
LazyValue and throws

    ArgumentError: JSON nesting depth at byte position 513 exceeds the
    `maxdepth` limit of 512; pass a larger `maxdepth` keyword argument to
    parse deeper input

before recursing. `maxdepth` is a new lazy/parse keyword (default 512),
passed like `allownan` or `jsonlines`. On master the shallowest path
(a recursive struct) overflowed at ~2,000 levels, so 512 keeps ~4x
headroom while allowing any realistic document.

The depth and limit are Int32 fields placed in existing struct padding,
so LazyValue (80 bytes), LazyOptions (48), LazyObject/LazyArray (72) keep
their sizes and allocations are byte-for-byte unchanged.

Measured against master on 56 read cases (instructions retired per call,
which machine load does not distort): +0.4% to +2% on ordinary documents,
up to +5-6% on inputs made only of empty containers (the check is five
instructions per object/array entered). Cycle counts and wall-clock time
showed no difference outside measurement noise. The same 42 methods
compile at first call; the package image grows 0.3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.60%. Comparing base (4f3dfdc) to head (e04da60).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #498      +/-   ##
==========================================
+ Coverage   92.57%   92.60%   +0.03%     
==========================================
  Files           7        7              
  Lines        1927     1935       +8     
==========================================
+ Hits         1784     1792       +8     
  Misses        143      143              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

On 32-bit Julia `DepthBox(1)` holds an Int32 while JSON parses the number
as Int64, so comparing the two structs failed on the x86 CI job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant