Skip to content

inlines: avoid repeated scans across unrelated delimiters - #658

Open
hakarrr wants to merge 6 commits into
commonmark:masterfrom
hakarrr:master
Open

hakarrr wants to merge 6 commits into
commonmark:masterfrom
hakarrr:master

Conversation

@hakarrr

@hakarrr hakarrr commented Nov 10, 2026 •

Copy link
Copy Markdown

Fixes #521.

Avoid repeated scans across unrelated delimiter types: each delimiter also belongs to a same-character chain used for opener searches, while the original combined list still controls processing order and removes delimiters consumed by emphasis or link scopes.

Per the discussion below, matched quote pairs now seal their scope: delimiters between the pair are removed, so quotes and emphasis no longer overlap and the output stays representable as an AST (e.g. for parsers with quote AST nodes like pandoc). Emphasis that pairs entirely inside the quotes is unaffected. 'a _b' c_ d now renders as <p>‘a _b’ c_ d</p> instead of the previous crossing <p>‘a <em>b’ c</em> d</p>.

Adds two smart-mode pathological cases based on the issue's examples, plus output regressions. The Python helper can enable smart mode through both the executable and shared-library paths.

Validation on Linux, on the current head:

  • Release build with CFLAGS='-O2 -Werror': 5/5 CTest tests passed; all 27 pathological cases pass (both smart cases time out against the unchanged baseline).
  • 4,000 generated documents: byte-identical output to the previous head of this branch in both ordinary and smart modes.
  • git diff --check passed.

Tradeoff: two additional pointers per internal delimiter and four pointers per subject. No public API change; the smart-mode output change for crossing quote/emphasis input is the intended behavior change discussed below.

Windows, macOS, 32-bit builds and clang-tidy were not tested locally.

hakarrr and others added 4 commits November 11, 2026 04:47
Keep the original delimiter ordering and smart-quote/emphasis semantics.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
@jgm

jgm commented Nov 10, 2026

Copy link
Copy Markdown
Member

This preserves crossing quotes and emphasis, including 'a b' c d, rather than changing precedence by removing all delimiters between matched quotes.

Not sure we'd want this. It works fine in cmark, but if we wanted to extend the syntax to other commonmark parsers that have AST nodes for quotes (e.g. pandoc), this would be unrepresentable. Not sure I want to set that precedent and not sure what the value is.

Avoid prescribing quote/emphasis precedence while that design remains open.

Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
@hakarrr

hakarrr commented Nov 10, 2026

Copy link
Copy Markdown
Author

That makes sense. I intended to preserve existing cmark output, not propose a cross-parser syntax rule. The unchanged base already renders 'a _b' c_ d as <p>‘a <em>b’ c</em> d</p>; this patch takes the separate-lists option from #521 to avoid the repeated scans without changing that output.

The new test wording was too prescriptive. I've changed it to describe current cmark output and clarified the PR description. No behavior or expected output changed; the updated head still passes 9/9 local tests.

Would you prefer the precedence-changing approach from #521, removing delimiters between matched quotes, instead of preserving current output? I'll hold off on a behavior change until that direction is clear.

@hakarrr

hakarrr commented Nov 13, 2026

Copy link
Copy Markdown
Author

Following up on the direction question above: would you prefer the precedence-changing approach from #521, or should this optimization remain limited to preserving current cmark output? If neither approach is wanted, please say so and I can close this. Thanks.

@jgm

jgm commented Nov 13, 2026

Copy link
Copy Markdown
Member

I think I'd like to keep existing behavior, for this change -- you reminded me that this is existing behavior, and of the discussion of #521 which I'd forgotten. Don't close the PR; it may take me some time to look at it.

@jgm

jgm commented Nov 13, 2026

Copy link
Copy Markdown
Member

@nwellnhof do you have opinions on this?

@nwellnhof

Copy link
Copy Markdown
Contributor

@nwellnhof do you have opinions on this?

Not a strong one, but it seems cleaner to end up with a solution that doesn't allow quotes or emphasis to overlap and results in something representable as AST.

@jgm

jgm commented Nov 14, 2026

Copy link
Copy Markdown
Member

OK, on consideration, I agree; let's not allow overlap.

Per the discussion on commonmark#658, drop the crossing quote/emphasis output
instead of preserving it: when a quote pair matches, remove the
delimiters between the pair so they can no longer match anything
outside it.  Quotes and emphasis then never overlap, and the result
stays representable as an AST.

Emphasis that pairs entirely inside matched quotes is unaffected,
since it is processed before the enclosing quote closer.

Co-authored-by: GeneralTony007 <144252170+GeneralTony007@users.noreply.github.com>
@hakarrr

hakarrr commented Nov 15, 2026

Copy link
Copy Markdown
Author

Reworked along the lines you and @nwellnhof settled on: matched quote pairs now seal their scope. When a quote pair matches, the delimiters between the pair are removed, so quotes and emphasis no longer overlap and the output stays representable as an AST. Emphasis that pairs entirely inside the quotes is unaffected, since it is processed before the enclosing quote closer.

'a _b' c_ d now renders as <p>‘a _b’ c_ d</p> instead of the crossing <p>‘a <em>b’ c</em> d</p>. The regression cases are flipped to the new semantics, and the PR description is updated.

On the current head (bfab5d9): 5/5 CTest, all 27 pathological cases pass (both smart cases still time out against the unchanged baseline), and a 4,000-document differential against the previous head of this branch is byte-identical in ordinary and smart modes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Quadratic behavior when parsing smart quotes

3 participants