[security disclosure] Unbounded Recursion in unpackValue() — Recoverable StackOverflow DoS #1015

An unauthenticated remote attacker sends a small payload of deeply nested fixarray[1] structures to trigger a StackOverflowError in the deserializing thread. StackOverflowError extends Error, so catch (IOException) and catch (MessagePackException) do not intercept it and the request fails. Availability is Low: the error is recoverable, the worker thread stays alive, and a top-level catch (Throwable) survives it. No data is read or modified.

Impact

Denial of service via a recoverable per-request deserialization failure.

• Escapes typed catch blocks: catch (IOException) / catch (MessagePackException) miss the StackOverflowError.
• Recoverable: the worker thread survives, the stack unwinds cleanly, and frameworks with catch (Throwable) recover fully. Impact is a per-request failure, not thread-pool exhaustion.
• Minimal payload: crash depth ≈1,500 with -Xss512k (1,501-byte payload) and ≈11,000 with the default stack (≈11 KB payload), both within typical HTTP body limits.

Remediation

Option 1 — Add maxNestingDepth to UnpackerConfig (recommended):

private int maxNestingDepth = 512; // UnpackerConfig field

public ImmutableValue unpackValue() throws IOException {
    return unpackValue(0);
}

private ImmutableValue unpackValue(int depth) throws IOException {
    if (depth > config.getMaxNestingDepth()) {
        throw new MessageSizeException("Nesting depth exceeds limit: " + depth, depth);
    }
    // ARRAY/MAP cases call unpackValue(depth + 1)
}

Option 2 — Convert to an iterative implementation using an explicit Deque stack, removing JVM-stack usage for nesting.

References

• msgpack-core/src/main/java/org/msgpack/core/MessageUnpacker.java, lines 646–663
• CWE-674: https://cwe.mitre.org/data/definitions/674.html
• MessagePack array format: https://github.com/msgpack/msgpack/blob/master/spec.md#array-format-family
• Similar issue: CVE-2023-1370 (json-smart, unbounded nesting → StackOverflow DoS)