Error generated for Chinese when input includes SIP characters

[tventimi]

The Chinese module of ScriptShifter generates an error when the input includes character in the SIP (i.e. those with a codepoint in the range U+2xxxxx). Two examples of characters that generate such an error are

2A56E 𪕣 hu
24656 𤙖 xin

The error text is:

WARNING:
An internal error occurred.
If the error persists, contact the technical support team.

[scossu]

Scriptshifter is throwing an internal error on all Unicode characters I tested with a value higher than U+9999. I am only able to reproduce it with Chinese but it might be a more general issue.

[tventimi]

Hi Stefano,

Did you mean values higher than U+FFFF? From what I can see, it seems to do fine with characters in the BMP, e.g. Korean 뀀 U+B000. We've used SS for Korean and haven't experienced any issues with the Hangul range U+AC00-U+D7AF.

[scossu]

You are correct, SS has no problems with characters up to U+FFFF. I am investigating this.

With Korean there is another subtle problem:

echo '🤘' | ./sscli trans korean_nonames 2> /dev/null
ch'5al

The Korean logic applies code point offsets and that seems to lead to unexpected results for characters outside a certain range . I am still looking what that range is. I will tackle that prblem separately.

[scossu]

I found out the root cause of this issue.

The correct YAML notation for Unicode characters above U+FFFF is \Uxxxxxxxxx, that is, a backslash followed by a capital U and a zero-padded, eight-character long hex value. So, the sample values you provided, would be written as:

"\U0002A56E": "hu "
"\U00024656": "xin "

The former notation (\u2A56E was being interpreted as \u2A56 + E), thus becoming a two-character string and causing all sorts of problems.

I am correcting all the occurrences of this issue in _chinese_base.yml. I don't know if we have other mappings in the upper ranges yet. I can run some text searches.

I noticed that the first character you pasted is U+2A563 (which is a CJK character but not mapped in the Chinese table), not U+2A56E. The second one ("xin") transliterates correctly after some local tests.

@RandyBarry you may be interested in this note, too.

[scossu]

It looks like it's not easy to programmatically scan for errors in other tables. E.g. in the Urdu table:

"d\u0323d\u0323": "\u0688\u0651"

This is in a Roman to script section, where a Unicode character is followed by a d.

I have to double-check that the changes I made to Chinese are not affecting the Roman part. Chinese is easier in that it only has Script-to-Roman, so I can restrict the change to everything before the colon in each line (unless I get bad luck and there is a colon in some of the affected tokens...) The change would affect over 17,000 lines, so validating by hand is not an option.

[RandyBarry]

Yes, I’m very interested to know how Unicode characters beyond the BMP are encoded. I hadn’t had occasion to specify one but now I know. I hope I remember this detail the day I need to add one.Randy Barry - ᠷᠧᠨᠳᠢ ***@***.***>On Dec 4, 2025, at 22:45, Stefano Cossu ***@***.***> wrote:scossu left a comment (lcnetdev/scriptshifter#271) I found out the root cause of this issue. The correct YAML notation for Unicode characters above U+FFFF is \Uxxxxxxxxx, that is, a backslash followed by a capital U and a zero-padded, eight-character long hex value. So, the sample values you provided, would be written as: "\U0002A56E": "hu " "\U00024656": "xin " The former notation (\u2A56E was being interpreted as \u2A56 + E), thus becoming a two-character string and causing all sorts of problems. I am correcting all the occurrences of this issue in _chinese_base.yml. I don't know if we have other mappings in the upper ranges yet. I can run some text searches. I noticed that the first character you pasted is U+2A563 (which is a CJK character but not mapped in the Chinese table), not U+2A56E. The second one ("xin") transliterates correctly after some local tests. @RandyBarry you may be interested in this note, too. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[RandyBarry]

Could you capture the whole array, sort them then scan visually for the longer nest the end?Randy Barry - ᠷᠧᠨᠳᠢ ***@***.***>On Dec 4, 2025, at 23:00, Stefano Cossu ***@***.***> wrote:scossu left a comment (lcnetdev/scriptshifter#271) It looks like it's not easy to programmatically scan for errors in other tables. E.g. in the Urdu table: "d\u0323d\u0323": "\u0688\u0651" This is in a Roman to script section, where a Unicode character is followed by a d. I have to double-check that the changes I made to Chinese are not affecting the Roman part. Chinese is easier in that it only has Script-to-Roman, so I can restrict the change to everything before the colon in each line (unless I get bad luck and there is a colon in some of the affected tokens...) The change would affect over 17,000 lines, so validating by hand is not an option. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[scossu]

I verified the changes and they seem pretty safe to me. None of them were on the Roman side, or in the initial part where common names are listed that could contain a mix of Chinese and ASCII characters. That said, @tventimi you may want to test thoroughly before deploying this to production.

[scossu]

Fixed in #277.

[tventimi]

Thanks, Stefano, I tested a selection of SIP Chinese characters on the dev site, and confirmed they were processed correctly. (Even for characters not found in the dictionary, SS simply returned the unconverted character as expected, rather than generating an error.)