Skip to content

Stop the lexer reading past the end of a byte_range - #3040

Open
ksss wants to merge 1 commit into
ruby:masterfrom
ksss:ksss/lexer-end-pos
Open

Stop the lexer reading past the end of a byte_range#3040
ksss wants to merge 1 commit into
ruby:masterfrom
ksss:ksss/lexer-end-pos

Conversation

@ksss

@ksss ksss commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

rbs_next_char ends the input when byte_pos == end_pos. rbs_skip advances by a whole character, so a multibyte character starting before end_pos and ending after it steps over the boundary — equality never holds again, and the lexer reads to the end of the string.

The realistic way to land inside a character is to pass a character offset where a byte offset is expected — the mistake #2945 fixed in parse_inline_*_annotation. "日本語" is 5 characters but 11 bytes, so offset 5 falls inside , which spans bytes 4...7:

RBS::Parser.parse_type('"日本語" | Integer', byte_range: 0...5)
#=> RBS::Types::Union spanning the whole input, rather than an error

require_eof: true does not catch it, because by then the lexer really is at EOF.

It takes a character straddling the boundary, so ASCII-only input never hits it. On master today it also needs that character to sit somewhere non-ASCII is already accepted — a string literal or a comment — since a non-ASCII byte in an identifier stops at an ErrorToken first. #3028 lets identifiers hold non-ASCII, which widens the set of inputs that reach it, but the bug is independent of that PR.

Fix

Compare with >= so stepping over the boundary still ends the input.

That leaves the straddling character itself read whole, one character past the range. On master that is not observable — every case I could construct still ends in a parse error — so nothing more is done here. It becomes observable once an identifier can hold non-ASCII, since an identifier is a complete type on its own; #3028 tightens the bound there, where a test can cover it.

Test plan

  • rake test — 968 tests, 0 failures. Two errors also present on master, unrelated.
  • New test_parse__byte_range_ending_mid_character, confirmed to fail with the == comparison restored. Covers the character-offset mistake above, the byte offset the caller meant, and an end_pos past the end of the buffer.

`rbs_next_char` ends the input when `byte_pos == end_pos`. `rbs_skip`
advances by a whole character, so a multibyte character starting before
`end_pos` and ending after it steps over the boundary and equality never
holds again — the lexer then reads to the end of the string.

The realistic way to land inside a character is to pass a character
offset where a byte offset is expected, the mistake ruby#2945 fixed in
`parse_inline_*_annotation`. `"日本語"` is 5 characters but 11 bytes, so
offset 5 falls inside `本`:

    Parser.parse_type('"日本語" | Integer', byte_range: 0...5)
    #=> Types::Union spanning the whole input, rather than an error

`require_eof: true` does not catch it, because the lexer really is at
EOF by then. It needs a character to straddle the boundary, so
ASCII-only input never hits it.

Compare with `>=` so stepping over the boundary still ends the input.
@ksss
ksss force-pushed the ksss/lexer-end-pos branch from 7becc4e to 8df3c79 Compare July 28, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant