[fix](be) Harden JNI reader split lifecycle

[Gabriel39]

What problem does this PR solve?

This PR fixes all JNI-related To Do items under the DORIS-27038 epic:

• DORIS-27053: keep Paimon C++ splits on the legacy scanner until FileScannerV2 can dispatch them.
• DORIS-27058: honor query cancellation while JNI batches are fully filtered.
• DORIS-27059: make JNI open/close cleanup exception-safe and avoid marking a failed close complete.
• DORIS-27064: propagate adaptive batch-size updates to open JNI scanners and Paimon/Hudi hybrid children.
• DORIS-27077: preserve end-of-split state so repeated reads after scanner close are idempotent.
• DORIS-27078: retain split-level Paimon options and Hadoop configuration as rolling-upgrade fallbacks.

Issue Number: DORIS-27053, DORIS-27058, DORIS-27059, DORIS-27064, DORIS-27077, DORIS-27078

Related PR: N/A

Release note

Harden JNI table-reader lifecycle handling and rolling-upgrade compatibility for Paimon and Hudi scans.

Check List

• Added comments for scanner-selection, lifecycle, adaptive-batch, and compatibility invariants.
• Added focused unit tests covering cancellation, EOF idempotence, cleanup failure, adaptive batching, scanner selection, and Paimon fallback precedence.
• Passed 45 focused ASAN BE unit tests from 7 suites.
• Passed build-support/check-format.sh and git diff --check.
• Ran changed-file clang-tidy; analysis is blocked by the existing toolchain missing stddef.h and pre-existing repository diagnostics.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at 5:19 PM.
Workflow run: https://github.com/apache/doris/actions/runs/29195932553

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[Gabriel39]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at 5:19 PM.
Workflow run: https://github.com/apache/doris/actions/runs/29196846018

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[hello-stephen]

TPC-H: Total hot run time: 30004 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4dd46c9839e89e245846a4a58308e8de5482540, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17707	4099	4333	4099
q2	2026	341	209	209
q3	10274	1417	850	850
q4	4686	471	339	339
q5	7499	879	564	564
q6	179	172	139	139
q7	771	905	625	625
q8	9317	1557	1569	1557
q9	5652	4438	4430	4430
q10	6757	1796	1564	1564
q11	511	349	317	317
q12	699	566	439	439
q13	18086	3399	2767	2767
q14	263	259	240	240
q15	q16	791	783	711	711
q17	1000	999	1054	999
q18	7483	5624	5630	5624
q19	1306	1214	1069	1069
q20	768	613	554	554
q21	6383	2896	2581	2581
q22	460	367	327	327
Total cold run time: 102618 ms
Total hot run time: 30004 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5080	4753	4897	4753
q2	318	320	213	213
q3	5084	5272	4566	4566
q4	2076	2133	1396	1396
q5	5016	4662	4765	4662
q6	224	187	136	136
q7	1863	1698	1580	1580
q8	2519	2354	2206	2206
q9	7751	7339	7255	7255
q10	4627	4585	4145	4145
q11	539	402	356	356
q12	736	740	530	530
q13	2953	3396	2712	2712
q14	274	284	249	249
q15	q16	685	695	600	600
q17	1297	1272	1274	1272
q18	7347	7040	6870	6870
q19	1117	1086	1041	1041
q20	2228	2186	1960	1960
q21	5313	4604	4489	4489
q22	531	473	415	415
Total cold run time: 57578 ms
Total hot run time: 51406 ms

[hello-stephen]

TPC-DS: Total hot run time: 180273 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4dd46c9839e89e245846a4a58308e8de5482540, data reload: false

query5	4310	646	481	481
query6	452	226	201	201
query7	4831	594	336	336
query8	350	195	175	175
query9	8762	4102	4078	4078
query10	457	356	310	310
query11	5960	2394	2135	2135
query12	160	103	99	99
query13	1252	602	409	409
query14	6205	5282	4970	4970
query14_1	4254	4261	4269	4261
query15	207	206	179	179
query16	1034	464	435	435
query17	923	742	560	560
query18	2469	458	344	344
query19	200	184	142	142
query20	115	106	103	103
query21	228	154	131	131
query22	13631	13530	13380	13380
query23	17379	16516	16159	16159
query23_1	16319	16353	16306	16306
query24	7572	1797	1300	1300
query24_1	1332	1311	1309	1309
query25	588	466	398	398
query26	1330	351	217	217
query27	2636	587	387	387
query28	4436	2035	2064	2035
query29	1092	645	503	503
query30	338	267	231	231
query31	1131	1093	984	984
query32	111	66	61	61
query33	536	331	268	268
query34	1179	1201	665	665
query35	787	789	659	659
query36	1411	1386	1205	1205
query37	152	111	95	95
query38	1899	1693	1650	1650
query39	929	935	876	876
query39_1	902	875	881	875
query40	254	165	141	141
query41	71	71	70	70
query42	95	95	93	93
query43	326	342	295	295
query44	1502	802	782	782
query45	205	188	181	181
query46	1092	1226	791	791
query47	2364	2365	2180	2180
query48	419	411	306	306
query49	601	434	323	323
query50	1130	415	331	331
query51	10978	10940	10902	10902
query52	89	90	77	77
query53	276	284	205	205
query54	309	261	228	228
query55	77	74	70	70
query56	313	316	291	291
query57	1414	1402	1314	1314
query58	297	278	255	255
query59	1594	1640	1464	1464
query60	326	282	269	269
query61	192	147	138	138
query62	702	640	581	581
query63	242	202	204	202
query64	2826	1056	866	866
query65	4880	4729	4775	4729
query66	1795	508	389	389
query67	29771	29409	29312	29312
query68	3105	1545	1035	1035
query69	398	306	260	260
query70	1033	960	954	954
query71	372	335	320	320
query72	3103	2771	2384	2384
query73	806	805	433	433
query74	5100	4980	4740	4740
query75	2621	2595	2233	2233
query76	2313	1190	764	764
query77	360	383	283	283
query78	12298	12432	11704	11704
query79	1380	1201	780	780
query80	1298	541	450	450
query81	527	315	274	274
query82	594	161	125	125
query83	403	323	295	295
query84	322	157	127	127
query85	976	641	509	509
query86	422	298	280	280
query87	1832	1821	1727	1727
query88	3727	2837	2790	2790
query89	454	405	359	359
query90	1932	203	201	201
query91	211	193	165	165
query92	65	69	55	55
query93	1572	1521	976	976
query94	721	363	324	324
query95	769	578	465	465
query96	1048	834	364	364
query97	2670	2676	2541	2541
query98	219	202	197	197
query99	1139	1185	1026	1026
Total cold run time: 266138 ms
Total hot run time: 180273 ms

[hello-stephen]

ClickBench: Total hot run time: 24.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4dd46c9839e89e245846a4a58308e8de5482540, data reload: false

query1	0.01	0.00	0.01
query2	0.10	0.05	0.05
query3	0.26	0.14	0.14
query4	1.61	0.14	0.14
query5	0.24	0.24	0.23
query6	1.34	1.06	1.09
query7	0.04	0.01	0.01
query8	0.06	0.03	0.04
query9	0.39	0.32	0.31
query10	0.60	0.56	0.56
query11	0.21	0.14	0.14
query12	0.19	0.15	0.14
query13	0.48	0.47	0.48
query14	1.02	1.00	1.00
query15	0.61	0.58	0.59
query16	0.32	0.33	0.31
query17	1.13	1.16	1.10
query18	0.23	0.22	0.21
query19	1.99	1.96	1.95
query20	0.02	0.01	0.02
query21	15.48	0.20	0.14
query22	4.96	0.05	0.05
query23	16.13	0.32	0.12
query24	2.95	0.44	0.35
query25	0.13	0.05	0.05
query26	0.73	0.21	0.14
query27	0.05	0.04	0.05
query28	3.58	0.92	0.51
query29	12.47	4.12	3.22
query30	0.28	0.16	0.18
query31	2.77	0.62	0.31
query32	3.23	0.59	0.49
query33	3.18	3.15	3.20
query34	15.46	4.27	3.51
query35	3.56	3.54	3.55
query36	0.55	0.46	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.04
query40	0.18	0.16	0.15
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.88 s
Total hot run time: 24.99 s

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 77.78% (84/108) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.72% (30167/40372)
Line Coverage	58.70% (331032/563963)
Region Coverage	55.44% (277594/500719)
Branch Coverage	56.73% (122775/216407)

[Gabriel39]

/review