Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: array_map could evaluate hidden nested values from outer-null array or map rows, and complete-offset validation could reject rows whose visible result should be NULL. Validate array lengths only for rows participating in lambda evaluation, rebuild only result offsets when NULL rows retain hidden payload, and read each argument through its own original offsets in the bounded execution path. Add direct BE unit coverage for hidden payload after valid rows, selector handling, multi-batch execution, per-input offsets, and non-NULL length mismatches.
### Release note
Fix lambda functions when nullable array or map rows retain hidden nested data.
### Check List (For Author)
- Test: Regression test and unit test
- Regression test: test_array_map_function, test_map_lambda
- Unit test: ArrayMapFunctionTest.* (30 tests)
- Static analysis: build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN
- Behavior changed: Yes. NULL container rows no longer participate in array-length validation or lambda evaluation.
- Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
There was a problem hiding this comment.
Static review completed against head 2b8e8a159abaa7888039f56919cf12e383bc9f2e. I am requesting changes for three blocking issues: the new compacted NULL-row offsets fail in several downstream multi-array consumers, lambda-form array_sort still evaluates hidden payload from NULL containers, and the required formatter check fails on the added unit tests.
Checkpoint conclusions:
- Goal and data correctness: the direct
array_mapbounded path and thetransform_valueswrapper are sound for the covered cases, but the broader null-container goal is incomplete because of the two behavioral issues above. - Scope and parallel paths: the production edit is focused. Selector mapping, per-input offsets, captures, batch boundaries, visible-row length checks, nullable/constant inputs, and direct/bounded/empty branches were checked. The sibling lambda-sort path and downstream scalar/aggregate consumers expose the reported gaps.
- Tests and expected results: the new unit/regression oracles are meaningful and the checked-in outputs match the SQL statically, but coverage omits the reported sort and mixed-row composition cases. The exact-head Clang Formatter job currently fails.
- Concurrency, lifecycle, configuration, compatibility, persistence, transactions/writes, FE-BE variable propagation, protocol, and observability: no applicable mechanism was changed and no separate issue was found.
- Memory/performance/error handling: no distinct memory-safety, ownership, status-propagation, or material performance defect was found in the changed core path.
This was a static review only: the review contract prohibited builds and test execution, so author/CI test claims were not independently rerun. There was no user-specified focus.
| const size_t row_size = current_offset - previous_offset; | ||
| compacted_rows += row_size; | ||
| } | ||
| res_offsets_data.push_back(compacted_rows); |
There was a problem hiding this comment.
[P1] Compacting an outer-NULL row to length 0 breaks callers that combine this result with another logically aligned array. In a mixed two-row block this result can have offsets [0,2] and null map [1,0] while the other array has [2,4]; nullable execution still exposes those physical offsets to the nested function. High-order array_split/array_reverse_split, array_zip(array_map(...), a), multi-argument array_enumerate_uniq, and multi-array _foreach aggregates then reject the unequal or shifted offsets instead of producing the required NULL row. Please make the nullable execution/affected consumers ignore or normalize outer-NULL rows, and add mixed NULL/non-NULL composition regressions.
| } | ||
| } | ||
|
|
||
| // NULL rows are skipped. If they retain hidden payload, rebuild only result offsets; |
There was a problem hiding this comment.
[P1] This skip is confined to ArrayMapFunction, but the parallel lambda-form array_sort path still evaluates hidden payload from an outer-NULL array. ArraySortFunction saves outside_null_map, then unconditionally std::sorts every original row range and invokes the comparator. With short_circuit_evaluation=false, a row such as array_sort((x,y) -> if(cast(x as int) < cast(y as int), -1, 1), if(id=1, cast(NULL as ARRAY<STRING>), values)) still raises on two hidden non-numeric strings even though the result row must be NULL. Please skip comparator execution for null rows too and add the corresponding strict-cast regression.
| auto nullable_array_int_type = std::make_shared<DataTypeNullable>(array_int_type); | ||
| std::vector<size_t> observed_batch_sizes; | ||
|
|
||
| auto root = |
There was a problem hiding this comment.
[P1] The required Clang Formatter check is failing on this new test block; the local clang-format 16 dry run reports violations from here through both added tests. Please run build-support/clang-format.sh on the changed C++ files and push the formatted result so the mandatory style gate can pass.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
With short-circuit evaluation disabled, array_map evaluated hidden nested values belonging to outer-null array or map rows. Invalid hidden values could therefore raise errors even though the visible result was NULL.
Compact nested lambda inputs for non-null rows and rebuild offsets before lambda execution, then preserve the original outer null map.
Release note
Fix lambda functions incorrectly evaluating nested values from NULL array and map containers.