GH-50338: [C++] Add ComputeLogicalNullCount to Datum#50347
Open
goel-skd wants to merge 2 commits into
Open
Conversation
|
|
goel-skd
force-pushed
the
gh-50338-datum-compute-logical-null-count
branch
from
2e82f92 to
30f92df
Compare
pitrou
reviewed
Jul 7, 2026
github-actions
Bot
added
awaiting committer review
goel-skd
commented
Jul 8, 2026
Comment on lines
-781
to
-817
| switch (dict_type.index_type()->id()) { | ||
| case Type::UINT8: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const UInt8Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::INT8: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const Int8Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::UINT16: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const UInt16Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::INT16: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const Int16Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::UINT32: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const UInt32Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::INT32: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const Int32Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::UINT64: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const UInt64Scalar&>(*value.index).value); | ||
| break; | ||
| case Type::INT64: | ||
| index_value = | ||
| static_cast<int64_t>(checked_cast<const Int64Scalar&>(*value.index).value); | ||
| break; | ||
| default: | ||
| return Status::TypeError("Not implemented dictionary index type"); | ||
| break; | ||
| } |
Contributor
Author
There was a problem hiding this comment.
Did a small refactor while I was here!
Datum::null_count() does not account for types that carry logical nulls without a validity bitmap (union, dictionary and run-end encoded types). Add Datum::ComputeLogicalNullCount(), delegating to ArrayData::ComputeLogicalNullCount() and ChunkedArray::ComputeLogicalNullCount() for array-like data; for scalars, is_valid already reflects logical validity.
goel-skd
force-pushed
the
gh-50338-datum-compute-logical-null-count
branch
from
f6dbdc7 to
ecb27c9
Compare
Contributor
Author
|
@pitrou - the s3fs test times out. Seems unrelated to this change and an CI infra red herring! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Datum::null_count()works on arrays, chunked arrays and scalars but ignores types whose logical nulls aren't in the top-level validity bitmap (union, run-end encoded, dictionary).ArrayData,ArraySpan,ArrayandChunkedArrayalready exposeComputeLogicalNullCount();Datumwas the missing piece.What changes are included in this PR?
Add
Datum::ComputeLogicalNullCount(), delegating toArrayData/ChunkedArray::ComputeLogicalNullCount()for array-like data and to a newScalar::IsLogicalNull()for scalars.Scalar::IsLogicalNull()returns!is_validfor most types (union and run-end encoded scalars already deriveis_validfrom their child).DictionaryScalaroverrides it so a valid index pointing at a null dictionary value counts as null, matchingdict_util::LogicalNullCount(); it reads the dictionary's validity directly (no encoded-scalar allocation) and falls back to!is_validfor ill-formed scalars. Like the array path, it does not recurse into nested (union/REE/extension-wrapped) values.Are these changes tested?
Yes.
Datum.ComputeLogicalNullCountcovers scalars,null()-typed array/scalar, a bitmap array, sparse union, chunked union, run-end encoded and dictionary arrays, plus scalars extracted viaArray::GetScalar().Scalar::IsLogicalNull()is also tested directly across all dictionary index types, including the out-of-bounds fallback.Are there any user-facing changes?
Adds
Datum::ComputeLogicalNullCount()andScalar::IsLogicalNull(); purely additive. The new virtual onScalaris a C++ ABI change (do not backport to a patch release).