Skip to content

[fix](lambda) Skip lambda evaluation for null containers - #68022

Open
linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:array-map-null-map
Open

linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:array-map-null-map

Conversation

@linrrzqqq

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

With short-circuit evaluation disabled, array_map evaluated hidden nested values belonging to outer-null array or map rows. Invalid hidden values could therefore raise errors even though the visible result was NULL.

Compact nested lambda inputs for non-null rows and rebuild offsets before lambda execution, then preserve the original outer null map.

CREATE TABLE test_array_map_null_filt (
    id INT,
    a ARRAY<STRING>
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES('replication_num'='1');

INSERT INTO test_array_map_null_filt VALUES
    (1, array('bad-number')),
    (2, array('10'));

SET enable_strict_cast = true;
SET short_circuit_evaluation = false;

SELECT
    id,
    array_map(x -> CAST(x AS INT), if(id = 1, CAST(NULL AS ARRAY<STRING>), a)) AS result_array
FROM test_array_map_null_filt
ORDER BY id;

SELECT
    id,
    array_map(x -> CAST(x AS INT), if(id = 1, CAST(NULL AS ARRAY<STRING>), a)) AS result_array
FROM test_array_map_null_filt
ORDER BY id;

-- before:
ERROR 1105 (HY000): Exception, msg: (127.0.0.1)[INVALID_ARGUMENT]parse number fail, string: 'bad-number'

-- now:
+------+--------------+
| id   | result_array |
+------+--------------+
|    1 | NULL         |
|    2 | [10]         |
+------+--------------+

Release note

Fix lambda functions incorrectly evaluating nested values from NULL array and map containers.

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: array_map could evaluate hidden nested values from outer-null array or map rows, and complete-offset validation could reject rows whose visible result should be NULL. Validate array lengths only for rows participating in lambda evaluation, rebuild only result offsets when NULL rows retain hidden payload, and read each argument through its own original offsets in the bounded execution path. Add direct BE unit coverage for hidden payload after valid rows, selector handling, multi-batch execution, per-input offsets, and non-NULL length mismatches.

### Release note

Fix lambda functions when nullable array or map rows retain hidden nested data.

### Check List (For Author)

- Test: Regression test and unit test
    - Regression test: test_array_map_function, test_map_lambda
    - Unit test: ArrayMapFunctionTest.* (30 tests)
    - Static analysis: build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN
- Behavior changed: Yes. NULL container rows no longer participate in array-length validation or lambda evaluation.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review completed against head 2b8e8a159abaa7888039f56919cf12e383bc9f2e. I am requesting changes for three blocking issues: the new compacted NULL-row offsets fail in several downstream multi-array consumers, lambda-form array_sort still evaluates hidden payload from NULL containers, and the required formatter check fails on the added unit tests.

Checkpoint conclusions:

  • Goal and data correctness: the direct array_map bounded path and the transform_values wrapper are sound for the covered cases, but the broader null-container goal is incomplete because of the two behavioral issues above.
  • Scope and parallel paths: the production edit is focused. Selector mapping, per-input offsets, captures, batch boundaries, visible-row length checks, nullable/constant inputs, and direct/bounded/empty branches were checked. The sibling lambda-sort path and downstream scalar/aggregate consumers expose the reported gaps.
  • Tests and expected results: the new unit/regression oracles are meaningful and the checked-in outputs match the SQL statically, but coverage omits the reported sort and mixed-row composition cases. The exact-head Clang Formatter job currently fails.
  • Concurrency, lifecycle, configuration, compatibility, persistence, transactions/writes, FE-BE variable propagation, protocol, and observability: no applicable mechanism was changed and no separate issue was found.
  • Memory/performance/error handling: no distinct memory-safety, ownership, status-propagation, or material performance defect was found in the changed core path.

This was a static review only: the review contract prohibited builds and test execution, so author/CI test claims were not independently rerun. There was no user-specified focus.

const size_t row_size = current_offset - previous_offset;
compacted_rows += row_size;
}
res_offsets_data.push_back(compacted_rows);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Compacting an outer-NULL row to length 0 breaks callers that combine this result with another logically aligned array. In a mixed two-row block this result can have offsets [0,2] and null map [1,0] while the other array has [2,4]; nullable execution still exposes those physical offsets to the nested function. High-order array_split/array_reverse_split, array_zip(array_map(...), a), multi-argument array_enumerate_uniq, and multi-array _foreach aggregates then reject the unequal or shifted offsets instead of producing the required NULL row. Please make the nullable execution/affected consumers ignore or normalize outer-NULL rows, and add mixed NULL/non-NULL composition regressions.

}
}

// NULL rows are skipped. If they retain hidden payload, rebuild only result offsets;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] This skip is confined to ArrayMapFunction, but the parallel lambda-form array_sort path still evaluates hidden payload from an outer-NULL array. ArraySortFunction saves outside_null_map, then unconditionally std::sorts every original row range and invokes the comparator. With short_circuit_evaluation=false, a row such as array_sort((x,y) -> if(cast(x as int) < cast(y as int), -1, 1), if(id=1, cast(NULL as ARRAY<STRING>), values)) still raises on two hidden non-numeric strings even though the result row must be NULL. Please skip comparator execution for null rows too and add the corresponding strict-cast regression.

auto nullable_array_int_type = std::make_shared<DataTypeNullable>(array_int_type);
std::vector<size_t> observed_batch_sizes;

auto root =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] The required Clang Formatter check is failing on this new test block; the local clang-format 16 dry run reports violations from here through both added tests. Please run build-support/clang-format.sh on the changed C++ files and push the formatted result so the mandatory style gate can pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants