[python] Build global index shards concurrently

[TheR1sing3un]

Purpose

PyPaimon's generic global-index builder currently builds vector and full-text shards one at a time. On multi-shard builds this leaves storage bandwidth and native index capacity idle.

Add global-index.build.parallelism (default 1) and use a bounded thread pool when the value is greater than one. Each worker keeps the existing streaming Arrow reader and one writer per shard, manifest messages are returned in shard-plan order, and a failed build closes active resources and removes every uncommitted index file. The documentation warns that each shard may also start native worker threads.

Benchmark

A temporary local benchmark exercised the production shard scheduler and streaming batch path with 4,000 approximately 2 KiB text rows per shard. It injected 20 ms of read latency plus 40 ms of native-build/upload latency per shard to isolate the scheduling change. Results are medians of three runs on macOS arm64; RSS is peak resident memory.

Shards	Parallelism	Time (s)	Speedup	Peak RSS (MiB)
4	1	0.448	1.00x	70.2
4	2	0.287	1.56x	99.6
4	4	0.207	2.16x	150.3
8	1	0.763	1.00x	70.2
8	2	0.456	1.68x	99.5
8	4	0.294	2.59x	157.7
8	8	0.238	3.21x	269.3
16	1	1.421	1.00x	70.2
16	2	0.783	1.82x	99.7
16	4	0.470	3.02x	157.8
16	8	0.339	4.20x	274.4

The ablation shows the throughput/memory tradeoff behind the conservative default. The benchmark uses simulated native and I/O latency, so end-to-end gains will depend on the index implementation, storage, shard size, and native thread settings. The temporary benchmark harness is not included in this change.

Tests

• python -m pytest -q pypaimon/tests/global_index_build_test.py (36 passed)
• python -m flake8 --config=dev/cfg.ini pypaimon/common/options/core_options.py pypaimon/globalindex/create_global_index.py pypaimon/tests/global_index_build_test.py