What is the problem the feature request solves?
Comet's native Iceberg scan never passes iceberg-rust a predicate on a decimal column. IcebergReflection.pageIndexUnsupportedColumns lists every decimal, uuid, fixed and binary column. CometIcebergNativeScan.icebergExprToProto then drops any residual predicate that references one, including the IS NOT NULL that Iceberg adds for every filtered column.
Results stay correct. Iceberg's planning still prunes files on manifest statistics, and the filter runs after the scan. But the native reader never prunes row groups or pages on a decimal column. A selective decimal filter therefore reads every row group and page of every file that survives planning.
#4982 added the gate because iceberg-rust, at the pins before #6649, could not handle these columns in the page index:
None of these holds at the current pins:
The Scaladoc on pageIndexUnsupportedColumns still describes the behavior from before #6649.
Describe the potential solution
Push decimal predicates through to iceberg-rust:
CometIcebergNativeScan.predicateLiteralToProto:
- Build
IcebergLiteral.decimal_val for a DecimalType column, at the column's precision and scale.
- If a literal can't be represented exactly at that scale, drop that predicate on its own. A literal that fails to bind would make iceberg-rust drop the whole residual.
iceberg_literal_to_datum in planner.rs: turn DecimalVal into a Datum with Datum::decimal_with_precision. iceberg-rust's Decimal is now fastnum::D128, so all 38 digits fit.IcebergReflection.pageIndexUnsupportedColumns: stop listing decimals.- Update three comments that say decimal pushdown is deferred: the Scaladoc on
pageIndexUnsupportedColumns, the doc comment on iceberg_literal_to_datum, and the comment on IcebergDecimal in operator.proto.
What works at which pin:
The other gated types:
uuid and fixed should stay in the list. iceberg-rust still skips their page indexes, and a uuid predicate's string literal doesn't bind to a uuid field.binary could come out too, but there is no binary predicate literal, so only IS [NOT] NULL would gain anything.
Tests should compare against Spark with checkIcebergNativeScan. The data should be decimal columns at precisions 9, 18 and 38, written in order over many small pages (write.parquet.page-row-limit). Cover:
- values on both sides of zero;
- range, equality and
IN predicates;
- a literal at a different scale from the column.
A pruned page that held a match loses rows silently, so these comparisons are the main guard. A check that the scan actually prunes, for example on the scan's bytes_scanned metric, would show the change has an effect.
Additional context
Found while reviewing #6851. A probe on that branch wrote a sorted decimal(38, 2) column over 200 pages, from -10100.00 to 10098.99, and ran nine predicates through the native scan. All nine matched Spark. But they only exercised the filter that runs after the scan, because the gate kept the predicates away from iceberg-rust.
What is the problem the feature request solves?
Comet's native Iceberg scan never passes iceberg-rust a predicate on a decimal column.
IcebergReflection.pageIndexUnsupportedColumnslists everydecimal,uuid,fixedandbinarycolumn.CometIcebergNativeScan.icebergExprToProtothen drops any residual predicate that references one, including theIS NOT NULLthat Iceberg adds for every filtered column.Results stay correct. Iceberg's planning still prunes files on manifest statistics, and the filter runs after the scan. But the native reader never prunes row groups or pages on a decimal column. A selective decimal filter therefore reads every row group and page of every file that survives planning.
#4982 added the gate because iceberg-rust, at the pins before #6649, could not handle these columns in the page index:
FIXED_LEN_BYTE_ARRAYcolumn index failed the scan withFeatureUnsupported. That layout backsuuid,fixed, and decimals of precision 19 and above.binarypage bound that was not UTF-8 panicked inString::from_utf8(..).unwrap().INT32orINT64decimal (precision 18 and below) could drop pages that held matching rows (fix(reader): page-index pruning drops rows for INT32/INT64 decimals iceberg-rust#3314).None of these holds at the current pins:
mainhas been at af1da4c since fix: make the native Iceberg per-I/O timeout configurable #6649, and that pin includes fix(reader): page-index pruning drops rows for INT32/INT64 decimals iceberg-rust#3314. It skips aFIXED_LEN_BYTE_ARRAYorINT96column index instead of failing, and it decodesbinarybounds as bytes.FIXED_LEN_BYTE_ARRAYdecimals (feat(reader): page-index pruning for FIXED_LEN_BYTE_ARRAY decimals iceberg-rust#3328).The Scaladoc on
pageIndexUnsupportedColumnsstill describes the behavior from before #6649.Describe the potential solution
Push decimal predicates through to iceberg-rust:
CometIcebergNativeScan.predicateLiteralToProto:IcebergLiteral.decimal_valfor aDecimalTypecolumn, at the column's precision and scale.iceberg_literal_to_datuminplanner.rs: turnDecimalValinto aDatumwithDatum::decimal_with_precision. iceberg-rust'sDecimalis nowfastnum::D128, so all 38 digits fit.IcebergReflection.pageIndexUnsupportedColumns: stop listing decimals.pageIndexUnsupportedColumns, the doc comment oniceberg_literal_to_datum, and the comment onIcebergDecimalinoperator.proto.What works at which pin:
main's pin: page pruning for decimals of precision 18 and below, and row-group pruning for every precision.The other gated types:
uuidandfixedshould stay in the list. iceberg-rust still skips their page indexes, and auuidpredicate's string literal doesn't bind to auuidfield.binarycould come out too, but there is no binary predicate literal, so onlyIS [NOT] NULLwould gain anything.Tests should compare against Spark with
checkIcebergNativeScan. The data should be decimal columns at precisions 9, 18 and 38, written in order over many small pages (write.parquet.page-row-limit). Cover:INpredicates;A pruned page that held a match loses rows silently, so these comparisons are the main guard. A check that the scan actually prunes, for example on the scan's
bytes_scannedmetric, would show the change has an effect.Additional context
Found while reviewing #6851. A probe on that branch wrote a sorted
decimal(38, 2)column over 200 pages, from -10100.00 to 10098.99, and ran nine predicates through the native scan. All nine matched Spark. But they only exercised the filter that runs after the scan, because the gate kept the predicates away from iceberg-rust.