Remove spaces from newlines between CJK characters

[isuffix]

This PR makes spaces due to newlines between CJK characters collapse to avoid creating a space when rendered. I've done so by splitting the Space syntax kind into two kinds to determine whether a space had a newline, and then by modifying the space-collapsing algorithm during Typst's realization step.

This is the behavior I mentioned in #792 (comment) (I have since changed my username from wrzian to isuffix).

Closes #792

Besides the below tradeoffs, I also have a few TODO comments that I would like input on.

Tradeoffs

This is a robust solution, but it makes multiple choices with tradeoffs that I'm not sure are desireable. I do not speak any CJK languages, so I'd appreciate feedback from the community about what is/isn't desired :)
CC @peng1999, @YDX-2147483647, @account-login

The main alternative design would be to solely resolve this in the parser or the AST, like YDX mentions in #792 (comment). That may improve or harm any of the tradeoffs below based on your opinion.

Each of the tradeoffs is exemplified in one or more new test cases:

Tradeoff: Collapsing happens during realization

Because collapsing happens after realization, a single newline space in the document may or may not collapse if its neighbors evaluate to CJK/non-CJK text dynamically.

Additionally, since a space element can itself be stored in a variable, static CJK characters can have different spacing due to stored space variables.

This is obviously one of the more contentious tradeoffs, but I think it's mostly fine. The first case is reasonable, and I doubt the second case is likely to affect real documents. But I am ok with changing either.

Test and output: space-collapsing-cjk-dynamic

--- space-collapsing-cjk-dynamic ---
// Test cjk space collapsing with dynamic variables.
#let foo = [水果] // collapses
#foo
#foo

#let foo = [fruit] // doesn't collapse
#foo
#foo

#let one-newline = [
]
#let no-newline = [ ]
啊#one-newline;啊 // collapses

啊#no-newline;啊 // doesn't collapse

Tradeoff: Normal spaces collapse only if they are adjacent to a newline space

Spaces that are not from newlines are kept, as in 空 格 (see the test case dropdown), but when adjacent to a newline space they will collapse.

While the basic behavior of collapsing a newline space or a newline space followed by a comment are straightforward, it's less clear whether a normal space followed by a comment and a newline space should combine as one space and collapse together, or if they should act separately. In addition, it's unclear if spaces with different styling should be able to combine and collapse together.

Currently, all three of the cases in this codeblock will combine and collapse their spaces. For me, I think the first is good, and the second is probably desired, but I'm less sure about the third case. (these are rendered in the dropdown below)

注//
释

空格 //
注释

*空格 *
换行

Tests and output: issue-792-space-collapsing-cjk and space-collapsing-cjk-strong

--- issue-792-space-collapsing-cjk ---
// Test how spaces with/without newlines collapse in CJK text.

// No space from just a newline/comments
换
行

注//
释

多行/*
*/注释

// Should have a space from a space character
空 格

// With both a space and a newline it still collapses
空格 //
注释

--- space-collapsing-cjk-strong ---
// Test cjk space collapsing with strong emphasis.
*空 *格

*换*
行

// This space still collapses because it is followed by a newline space.
*空格 *
换行

Tradeoff: Treating space values as equal

This is the one I'm least certain about, and would be improved by ignoring spaces in the parser/AST.

There are a few other test cases (list-indent-trivia-nesting, list-indent-bracket-nesting) that implicitly expect space elements to be equal to each other regardless of newlines. I feel like I also generally expect this behavior (I wrote those tetsts), so breaking them feels odd. I added a custom PartialEq implementation to SpaceElem that always returns true to make this work. However, I'm not sure if this is sound with the way Comemo caches data.

I'm totally ok with removing this if we stay with space-collapsing during realization, but it will require modifying the other test cases if we do, and we would probably also want to modify the repr of SpaceElem.

Test: space-eq-newline

--- space-eq-newline ---
// Test whether spaces with/without newlines compare equal.
#let parbreak = [

]
#let one-newline = [
]
#let no-newline = [ ]
// parbreak is not equal
#assert.ne(one-newline, parbreak)
// spaces are equal despite newlines
// TODO: Would this break comemo?
#assert.eq(one-newline, no-newline)

[YDX-2147483647]

Thank you for implementing this!

As for space-collapsing-cjk-strong (*空格 *), I don't think it's very important, because I guess no one would write it like this.
As for other aspects, I can take a closer look when I have time.

Besides, the cjk-unbreak package has accumulated a few test cases in https://github.com/KZNS/cjk-unbreak/blob/2ea9b0ce3654ab537116499f63aa4077165192bc/test.typ. They might be relevant.

Update on 2025-12-17: The cjk-spacer package published recently is also relevant.
The two packages are developed by Chinese and Japanese people respectively. I guess there might be subtle differences, although I haven't compared them carefully yet.

[YDX-2147483647]

I suggest we also include punctuation marks, for the following use case.

Typst source:

我就站住，
豫备她来讨钱。
“你回来了？”
她先这样问。

Result:

我就站住，豫备她来讨钱。“你回来了？”她先这样问。

The list of CJK punctuation marks can be found in clreq and jlreq. (I don't know if K should be included here.)

This might be more complicated than it seems to be. For example, the following three characters are widely used in Chinese documents, but their categories are different. See regex.pdf for further comparisons.

• U+3001 、 顿号 (secondary comma) matches \p{Script_Extensions=Han}.

• U+FF0C ， 逗号 (regular full-width comma) matches \p{Script_Extensions=Common}.

• U+201C “ 上双引号 (left double quotation mark) matches \p{Script_Extensions=Common}, and it is also used in Latin documents.

They all match \p{General_Category=Punctuation} and \p{Script=Common}.

[ponte-vecchio]

For Korean, they should be included, along with its half-width counter parts of the Latin punctuations where possible---both of which are used.

[peng1999]

Please don't include Hangul here. Modern Korean uses spaces to separate words, so it's not relavant here. The function is better named as is_cj.

I used 'CJK' in the title of #792, that was a mistake. This issue only affects Chinese and Japanese.

[isuffix]

Thank you for the links, they're very helpful!

I can change to just CJ removing Hangul. I'll plan to move the function out of the lexer and leave the lexer behavior untouched (unless that should change too?)

However, I'd appreciate more thoughts on what to do for punctuation. It seems we have two categories of codepoints: non-ambiguous CJ punctuation and ambiguous CJ punctuation, such as left/right quotes. For non-ambiguous, I guess we can just treat them like all other CJ codepoints for collapsing, but I'm less sure what to do for ambiguous punctuation.

I presume it would be a good behavior for ambiguous punctuation followed by a CJ character (or vice-versa) to collapse a newline space, but what about an ambiguous punctuation next to another ambiguous punctuation? Are there any characters that would be likely to be split across lines? Should we look at the text language to determine this?

Also, some of the non-ambiguous CJ punctuation overlap in usage with Hangul. Should we be taking that into account?

This is also relevant for #5858, which has some related discussion around quotation marks.

[isuffix]

Another approach would be to just set the space collapsing behavior based on the text language, or an explicit property of say, par() (similar to the request in #710). This is more coarse-grained, but would simplify many of these considerations. Another tradeoff 😮‍💨

[tats-u]

@YDX-2147483647 According to https://www.unicode.org/Public/17.0.0/ucd/EastAsianWidth.txt, The EAW of U+17A4 is N, not W or F. Did you confuse it with something different?

[tats-u]

The Script property of U+115F is Hangul and EAW of it is W.

https://www.unicode.org/Public/17.0.0/ucd/Scripts.txt

[YDX-2147483647]

The EAW of U+17A4 is N, not W or F. Did you confuse it with something different?

Hi! I didn't make it clear. What I mean is as follows.

1. c.width() uses a complex rule to determine the width, and EAW is one of the factors.
2. The full rule is documented on https://docs.rs/unicode-width, and it says that U+17A4 and U+115F will give width 2. (For these two specific characters, EAW does not contribute to c.width().)
3. Therefore, I don't think c.width() == Some(2) is a good criterion.

[tats-u]

I agree with your opinion that unicode_with is not suitable for determining whether the character is CJ(K).
We should use the EAW property directly. Also, U+FF61 HALFWIDTH IDEOGRAPHIC FULL STOP is a (legacy) Japanese character but whose Script is not Katakana but Common and whose EAW is H. All non-Hangul/Korean characters whose EAW is H must be treated as Chinese/Japanese.

[tats-u]

If we have issues with determining whether characters are Korean, we should test their test cases in FIrefox and report them in https://bugzilla.mozilla.org/ (its bug tracker):

<div lang=ja><!-- ← or zh -->
  Test
  Case
  Here
</div>

Removing a newline around a punctuation is enabled only in Chinese and Japanese.

[tats-u]

biomejs/biome#7304 (comment)

Note: I personally prefer the Emoji_Presentation property to the Emoji property

[isuffix]

@tats-u thanks! The link to the CSS WG Draft is also helpful.

[tats-u]

The link to the w3c/csswg-drafts#5086 is also helpful.

It should be just a link to tracker. No concrete specification is stipulated in CSS WG Draft (https://drafts.csswg.org/css-text-4/#line-break-transform) now.

Prettier's issue (fixed):

• Prose wrap options for Korean prettier/prettier#6516

"K" should be excluded from the title, too.

FYI, the following JS expression returns [], which means that no "Korean-dedicated punctuation chracters (whose Script is Hangul and General Category is Punctuation)" exists:

Iterator.from((function*() {for (let i = 0; i <= 0x10ffff; i++) yield i;})()).filter(cp => /[\p{P}&&\p{sc=Hang}]/v.test(String.fromCodePoint(cp))).toArray()

[isuffix]

I rebased off main and have refactored the space collapsing algorithm quite a bit. It now both depends on #7609 and reifies the kinds of actions that the algorithm takes in a new enum. This shouldn't change any of this PR's high-level behavior (we still discard newline spaces if either side is a space-discarding character), but the new organization really helps me keep the full algorithm in my head.

Additionally, I've updated the space-discarding character set to include common-script characters only if their East Asian Width is F/W/H but they are not emoji (although I still anticipate that this can be improved). This is aided by the new test newline-space-discarding-edge-cases which lets us check characters at a very fine grained level.

To determine the East Asian Width, I've moved to using the icu_properties crate which seems to be the highest quality source of the property in Rust right now. I've also moved the Script and Emoji checks to use icu_properties for consistency. However, we are still inexplicably using version 1.4 while icu is on 2.0 since May (now 2.1), so I'd appreciate suggestions as to whether it should be updated (it does have breaking changes). If we did want to stick to crates from unicode-rs, I believe we could ask the maintainer to add the EAW to the unicode-properties crate. However, it was very annoying that I could not find any real guidance online as to whether we should prefer icu vs. unicode-rs.

[isuffix]

Here is the new space-discarding character check.

[tats-u]

There seems to be some other East Asian scripts that prefers without-space-style e.g. Yi.
Also you should add // Especially Hangul above _ => false,.

[OverflowCat]

There seems to be some other East Asian scripts that prefers without-space-style e.g. Yi.

So imo the feature is more about scripts that does not use spaces in writing than CJK characters only. This reminds me of Tangut -- don't know if they are Script::Han, though. Also, do we need to take those languages that do not use space within sentence level into account? e.g. Tibetan only uses spaces after a punctuation mark like ། .

[tats-u]

The advantage of relying only on EAW for non-Hangul scripts recognition is that it eliminates the need to worry about about which scripts must be covered, including minor scripts like Yi or Tangut.

[tats-u]

't know if they are Script::Han, though.

A dedicated Script Property Value Tangut is assigned to Tangut since Unicode 9 (2016).

https://www.unicode.org/standard/supported.html

[isuffix]

Let me know what characters I should add to this test and whether any should change.

[tats-u]

• ｱ U+FF71 true ←EAW is H
• ㊙ U+3299 ? (false in my opinion) ←This is a default text presentation character like ©. Without a succeeding U+FE0F, it should not be displayed as emoji if a proper Japanese font is assigned. However, the current Firefox treats both 2 symbols as emoji because it treats every emoji character (a character that has an Emoji property) as emoji. There is Emoji_Presentation property.
• 한 U+D55C false ←Prose wrap options for Korean prettier/prettier#6516
• ₩ U+20A9 ? (false in my opinion) ←EAW is H but Korea-dedicated. Since the EAW of ¥ U+00A5 for Japan and PRC is Na, ₩ should not be treated as Chinese-or-Japanese, either. Shamefully && c != '\u{20A9}' is needed.

It is recommended to also check the character two positions away as necessary to check emojiness of the adjacent grapheme more accurately.

[laurmaedje]

However, we are still inexplicably using version 1.4 while icu is on 2.0 since May (now 2.1), so I'd appreciate suggestions as to whether it should be updated (it does have breaking changes).

The reason is just that nobody got to it. But there is #7412. There's still some unresolved questions though.

However, it was very annoying that I could not find any real guidance online as to whether we should prefer icu vs. unicode-rs.

Definitely ICU.

[isuffix]

The first push rebases off the updated #7609, the second/third just add some of the mentioned edge case characters, although I haven't updated their implementation yet.

[isuffix]

I'd like to move this PR forward, but first—based on past discussions on Discord—we are going to move the definition for which characters should discard inferred spaces to the Codex crate.

I wrote an update in the Codex forge on Discord, and I would appreciate if any feedback could be posted there. But I am copying the message here for reference:

I'd like to push PR 7350 (CJ newline space discarding) over the finish line for initial inclusion in Typst. However, we first need to address how to add the character definitions for this behavior to Codex. I'm posting here to get feedback before I submit a new PR to Codex and update the existing PR.

I know that there was still concern about defining the feature as a static property on specific characters vs. basing this on a show/set rule or adding a general mechanism for customizing newlines in markup, but I still think that the current proposal makes for a sensible default that we can add further customization to in the future.

For reference, you may want to read my previous comment in the CJK forge along with Laurenz's response.

Adding to Codex

To add this to Codex I think it makes sense to create a new Cargo feature like numeral-systems and styling that we define in its own file under src. My current thought is to call the feature space-discarding, since the definition is intended to determine whether an inferred space due to a newline should be kept or discarded.

Please let me know if this feature name is fine or if you have any suggestions.

What will be added

As for what will be added under that feature, I'd still like to expose a simple function for whether to discard a newline space, but I think my original is_space_discarding(char) -> bool function has two problems that I'd like to fix.

The first problem is that we should really be considering grapheme clusters instead of codepoints, and I'd like the function to take &str instead of char to do so. The second problem is that what exactly the function is meant to represent feels ambiguous to me, and I think we should change how we define it to make it easier to reason about.

Taking char vs. &str

Instead of codepoints, what we really want to analyze are the grapheme clusters that surround the inferred space. This is because Unicode is complex and the immediately surrounding codepoints could be something like variation selectors that have a base codepoint that is what we actually want to make our determination based on. Accepting char as our argument would require every caller to implement that logic when it should really be the library which handles analyzing grapheme clusters.

And to avoid needing to expose two different functions for analyzing the initial/final grapheme clusters of a string, I think it would be best to have one function which accepts the two strings that come before and after the inferred space.

The new function would look like: discard_space_between(before: &str, after: &str) -> bool

I haven't actually implemented any grapheme cluster analysis yet (my current draft still just uses the first/last codepoint), but I think doing so is worthwhile and it's worth moving to &str now so that we can more easily add this in the future. And if we decide against doing any grapheme cluster analysis, changing from &str to char will likely be easier for users of the function than changing from char to &str.

Defining the function more precisely

I think only considering the high-level behavior of keeping or discarding spaces is likely to lead to ambiguity and confusing arguments. Instead of focusing the behavior of Typst itself, I think we should base the function on the relevant properties of the writing system/language the surrounding characters are from, and focus on mapping characters (or grapheme clusters) to the correct writing system/language. This would also make it easy to improve the function's accuracy by adding a lang parameter in the future to handle characters like puncuation whose writing system is ambiguous.

I think the property we should define and use is whether a writing system/language uses spaces between words. Chinese and Japanese dont, but Korean and many other writing systems do. Then the function would decide to discard an inferred space if either surrounding character is unambiguously from a writing system which does not uses spaces between words. This matches the current implementation in the PR, and I find it to be a reasonable default.

For the initial implementation I would leave it there, but I know that abugida-based writing systems such as Thai, Lao, Burmese, etc. use spaces between phrases and sentences but not between words. And I know other writing systems may also prefer different definitions, e.g. Vietnamese which uses spaces between syllables. But I think word-spacing is a good place to start and we can add more properties to account for other writing systems over time.

Please let me know if you have any feedback on this. If there are no concerns, I'll plan to submit this design as a PR to Codex and base PR 7350 on the new definition sometime next week.

[isuffix]

This is now based on typst/codex#182!

Since this updates the Codex dependency, I had to add a new commit to remove some deprecated symbol tests.