Add the space-discarding feature

[isuffix]

This adds the space-discarding feature as described at typst/typst#7350 (comment), although I have changed from "whether a writing system uses spaces between words" to "whether a writing system uses spaces at all."

I will leave this PR description short as the code itself contains a plenty of discussion of rationale and implementation considerations, along with my research into the usage of space characters in various writing systems.

There is a lot of writing here, so I would really appreciate help with checking for typos and inconsistencies, as it has become hard for me to consider everything with fresh eyes. I am very amenable to suggestions :)

I will also restate that I only speak English and while I have tried to do good research, I am not infallible. I would appreciate any input from native speakers of Chinese or Japanese or any of the other writing systems discussed in the PR.

I would also like to thank @r12a for his wonderfully detailed orthography descriptions and script comparison table, without which this PR would not be nearly as complete or authoritative. If you're reading this, I would love any feedback you could provide.

[r12a]

Tibetan comes to mind as an example of an orthography that doesn't use (ASCII) spaces but does have delimiters (syllable-based) which could break a line in the source code, but which should not incur an extra space when stitching things together (see Tibetan Orthography Notes). hth

[YDX-2147483647]

I've gone through the source code (except the emoji part). I believe the current algorithm is simple and robust enough. I only have some suggestions regarding doc comments and tests. See my individual comments.

More materials supporting the current algorithm

East Asian Width

UAX #14: Unicode Line Breaking Algorithm also uses East Asian Width to filter out East Asian characters.

The symbol $EastAsian stands for the set [\p{ea=F}\p{ea=W}\p{ea=H}] of characters with Fullwidth, Wide, or Halfwidth East Asian Width.

Pandoc

Pandoc's east_asian_line_breaks extension uses charWidth to determine if a soft break (e.g., single newline in markdown and typst) should be removed. However, we've argued in typst/typst#7350 (comment) that the rules for determining widths are too complicated and it's better to use East Asian Width directly.

LaTeX

LaTeX cannot be taken as a reference, because the implementations are limited by the technology. Specifically, the whitespace in 字\n“ should be discarded, but luatexja keeps it. And the whitespace in ”\nA should be kept as a word space, but xeCJK discards it. See typst/typst#792 (comment) for the tests.

Typst cjk-unbreak

As for typst packages, cjk-unbreak uses the following regex to determine if a character is CJ (Chinese + Japanese) and discards the space iff either side matches the regex.

[\p{Han}，。；：！？‘’“”（）「」【】…—\p{Hiragana}\p{Katakana}]

This package is designed only for CJ, so the regex includes a few characters that are considered YesOrAmbiguous in this PR.
The algorithm in this PR is designed for all writing systems, so the difference to cjk-unbreak is acceptable.

Typst cjk-spacer

A newer typst package, cjk-spacer, uses a more complex algorithm. If I understand correctly, then its algorithm is equivalent to the following.

#let default-cjk-regex = regex(
  "["
    + "\p{scx:Hira}\p{scx:Kana}\p{scx:Han}\p{scx:Hang}\p{scx:Bopo}"
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\u3190-\u319F" // Kanbun
    + "\u31C0-\u31EF" // CJK Strokes
    + "\u3200-\u32FF" // Enclosed CJK Letters and Months
    + "\u3300-\u33FF" // CJK Compatibility
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]",
)
#let default-western-open-punc-regex = regex(
  "[\p{Pi}\p{Ps}--["
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]]",
)
#let default-western-close-punc-regex = regex(
  "[\p{Pf}\p{Pe}\p{Term}--["
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]]",
)

#let discard_space_between(before, after) = {
  if after.matches(western-open-punc-regex).len() == 0 and after.starts-with(cjk-regex) {
    true
  } else if before.matches(western-close-punc-regex).len() == 0 and before.ends-with(cjk-regex) {
    true
  } else {
    false
  }
}

The cjk-spacer algorithm does not merely consider the characters immediately adjacent to the space, but rather the text segments around the space. This approach is appropriate when typesetting a document, but it's too surprising at the syntax level.

Also, cjk-spacer treats K the same as CJ. According to previous feedbacks in typst/typst#7350, discarding spaces is not preferable for Korean texts.

And I haven't check if the Unicode blocks enumerated by cjk-spacer are equivalent to this PR, but I think it's worth checking before merging this PR.

[YDX-2147483647]

The term ideographic punctuation needs clarification.

• There's a Unicode block called Ideographic Symbols and Punctuation, but obviously that isn't what you mean.

• ‼ is listed in both Punctuation marks in Chinese in CLReq and Character Classes in JLReq, but it isn't considered as a ideographic punctuation.

It looks like that you use this term colloquially. I suggest putting the relevant test cases in a separate function and linking to it. (similar to test_spacing_emoji_presentation)

[YDX-2147483647]

Suggested change

	check_spacing('⸺', YesOrAmbiguous); // two em-dash
	check_spacing('–', YesOrAmbiguous); // en dash
	check_spacing('—', YesOrAmbiguous); // em dash
	check_spacing('⸺', YesOrAmbiguous); // two-em dash

It should be two-em dash, not two em-dash.

[YDX-2147483647]

Suggested change

	/// We should avoid intoducing space characters that the author didn't intend
	/// We should avoid introducing space characters that the author didn't intend

[YDX-2147483647]

Suggested change

	/// `Wide` to determine common ideographic punctuation characters, unfortunately
	/// `Wide` to determine common ideographic punctuation characters. Unfortunately,

[YDX-2147483647]

Just a comment for your information: I get the narrow ¥ when I type in Chinese mode on my mobile phone, but I get the fullwidth ￥ when I press Shift+4 ($) in Chinese mode on my desktop computer.

[YDX-2147483647]

I think it's necessary to explain the meaning of discarding spaces here, as the word space is also used to refer to the lack of ink below in WritingSystemSpacing.

My suggestion:

• Refer to the lack of ink as spacing.
• Refer to U+0020 SPACE and \n as whitespace characters.

[YDX-2147483647]

Is there any use case for empty strings and spaces?

If there is, we should mention it in the public doc comments.
Otherwise, perhaps we should make discard_space_between(a, b) always equivalent to discard_space_between(a.trim(), b.trim()). That might be useful when dealing with the following markdown.

- 比如
  这样
- The second
  point

[YDX-2147483647]

Is it possible to replace these check_spacing(.., No) lines with something like the following? This can prevent writing_system_spacing from discarding spaces around punctuations accidentally.

assert_eq!(
  (0..=0x10FFFF)
    .filter_map(char::from_u32)
    .filter(|c| c.is_puncutation() && matches!(writing_system_spacing(c), No)),
  &[..],
);