Skip to content

avoid quadratic getToken(i) rescans in lookahead token walks - #2862

Merged
manticore-projects merged 1 commit into
JSQLParser:masterfrom
tanjiroK-coder:lookahead-token-walks
Oct 27, 2026
Merged

manticore-projects merged 1 commit into
JSQLParser:masterfrom
tanjiroK-coder:lookahead-token-walks

Conversation

@tanjiroK-coder

Copy link
Copy Markdown
Contributor

Several semantic lookahead helpers in JSqlParserCC.jjt scan forward with getToken(i) inside an unbounded loop, and getToken(i) walks i tokens from the current position on every call, so each scan is quadratic in the number of tokens it covers (the same problem #2780 fixed in isNestedSetOperationAhead). The worst case is the memoizing pass of isFunctionAheadWalk / isAllTableColumnsWalk, which any dotted name with more than eight parts reaches: SELECT a0.a1.....a7999 FROM t (47 KB) spends about 16.5 s in CCJSqlParserUtil.parse, well past the default 8 s timeout. I found it while profiling parse time on long generated statements, where the samples sat in isAllTableColumnsWalk on getToken. isTernaryAhead (probed from isComparisonOperatorAhead during lookahead), isObjectPrivilegeAhead, isStatementBlockAhead, isInsertOnConflictAhead, isPostgreSqlTriggerAhead, isNamedExprListAhead, isPostgreSqlRoutineCharacteristicAhead and the role loop in isMySqlSetRoleAhead use the same pattern, so they now follow Token.next through the existing nextWithItemToken() helper as well, visiting the same tokens in the same order. With the change the 8000-part chain parses in about 30 ms and 64000 parts (437 KB) in about 0.4 s, and a 16000-privilege GRANT drops from 4.4 s to 0.14 s. The new LongChainPredicateTest case parses 10000-part column, .* and function chains under a 5 s preemptive timeout; it fails on master and passes in 0.4 s here, and ./gradlew check is green apart from StringValueTest's timing test, which timed out once during the full run and passes on its own.

@manticore-projects
manticore-projects merged commit 6c71e15 into JSQLParser:master Oct 27, 2026
10 checks passed
@manticore-projects

Copy link
Copy Markdown
Contributor

Thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants