GH-3735: Use unsigned UTF-8 byte order for Variant object field keys

[peterxcli]

Rationale for this change

The Variant encoding specification requires the field ids in an object's header to be sorted by the unsigned byte order of the field names' UTF-8 encoding, so readers can binary-search them. VariantBuilder sorted the fields — and Variant.getFieldByKey binary-searched them — using String.compareTo, which orders UTF-16 code units instead.

The two orderings agree for all keys in the Basic Multilingual Plane but diverge for supplementary-plane characters (U+10000 and above): String.compareTo orders a leading high surrogate (0xD800–0xDBFF) before code points in U+E000..U+FFFF, whereas UTF-8 byte order (and the spec) orders them after. Consequences:

• Objects parquet-java builds with such keys have field ids sorted in a spec-violating order, so spec-compliant readers can fail to find fields via binary search.
• parquet-java's own binary search can fail to find a supplementary-plane key in an object produced by a spec-compliant writer.

This adapts #3736 by @rayokota and adds the read-compatibility fallback from the equivalent Spark fix (apache/spark#58239), per the discussion on that PR.

What changes are included in this PR?

• New VariantUtil.encodeKey(String) and VariantUtil.compareKeys(byte[], byte[]), which order field names by unsigned lexicographic UTF-8 byte order (from GH-3735 Fix Variant field name comparisons to use UTF-8 byte order #3736)
• VariantBuilder.FieldEntry.compareTo sorts object fields with that comparison, lazily caching each field's UTF-8 encoding (from GH-3735 Fix Variant field name comparisons to use UTF-8 byte order #3736)
• Variant.getFieldByKey binary-searches in UTF-8 byte order first; for keys containing a code unit at or above U+D800 (the only keys where the two orders can differ), it retries the search in UTF-16 order, so objects written by older versions in the legacy order remain readable

Are these changes tested?

Three new tests in TestVariantObjectBuilder:

• testObjectKeysSortedByUtf8ByteOrder — builds an object with keys U+FFFF (EF BF BF) and U+10000 (F0 90 80 80) appended in reverse and asserts the encoded field order is U+FFFF then U+10000 (UTF-8 order), which the previous compareTo reversed.
• testLargeObjectBinarySearchWithSupplementaryKey — a 42-field object (above BINARY_SEARCH_THRESHOLD) mixing ASCII keys with U+FFFF and U+10000, asserting getFieldByKey resolves both through the binary-search path.
• testLegacyUtf16OrderedObjectLookup — rewrites a canonical object's id and offset lists into the legacy UTF-16 order and asserts getFieldByKey still finds ASCII, U+FFFF, and U+10000 keys through the fallback search, and that absent keys stay absent.

All 183 parquet-variant tests and the parquet-avro variant read/write suites pass locally.

Are there any user-facing changes?

Newly written Variant objects containing supplementary-plane field keys now use the specification's unsigned UTF-8 field order. Objects written in the legacy UTF-16 order remain readable via getFieldByKey.

Closes #3735

🤖 Generated with Claude Code