Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .ai/skills/review-comet-iceberg-write-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,8 @@ only the latest.
that starts trusting iceberg-rust's metrics, or adds a native-tracked value beyond float and
double NaN counts and bounds, needs a strong reason.
- [ ] **Value semantics match, not only formats.** iceberg-rust compares and hashes partition values
with its own rules: `OrderedFloat` treats `-0.0` and `0.0` as equal where Java's
`Float.compare` does not
with its own rules: until apache/iceberg-rust#3327 its `OrderedFloat` equality treated
`-0.0` and `0.0` as equal where Java's `Float.compare` does not
([#6138](https://github.com/apache/datafusion-comet/issues/6138)). Check equality, hashing,
ordering and rendering for float, double, timestamp, timestamptz, binary and decimal.
- [ ] **Timestamp partition values are UTC.** Iceberg's `years`, `months`, `days` and `hours`
Expand Down
5 changes: 3 additions & 2 deletions docs/source/contributor-guide/iceberg-writes.md
Original file line number Diff line number Diff line change
Expand Up @@ -395,8 +395,9 @@ Useful places to look when checking parity:
- `CometIcebergWriteActionSuite` writes the same data through both writers into sibling tables and
compares rows, `readable_metrics` and partition paths.
- iceberg-rust's own code, at the pinned revision. iceberg-rust makes different choices from
iceberg-java in places that matter to the table's contents, for example grouping partition keys
with `OrderedFloat`, which treats `-0.0` and `0.0` as equal
iceberg-java in places that matter to the table's contents. For example, until
[apache/iceberg-rust#3327](https://github.com/apache/iceberg-rust/pull/3327) it grouped
partition keys with `OrderedFloat`, which treats `-0.0` and `0.0` as equal
([#6138](https://github.com/apache/datafusion-comet/issues/6138)). Check how iceberg-rust
compares, hashes and renders values, not only what it writes.

Expand Down
13 changes: 2 additions & 11 deletions docs/source/user-guide/latest/iceberg-writes.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ A write is eligible only when ALL of the following hold:
| resolved `table.locationProvider()` | Iceberg's built-in `DefaultLocationProvider` |
| Hadoop S3A settings for an `s3` / `s3a` data location | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back |
| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location | the S3 endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, `kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain settings parsed by the pinned iceberg-rust version, plus Comet's credential-provider class and built-in web-identity properties. When a custom provider is configured, its vendor-owned `s3.*` / `client.*` properties are also forwarded; unsupported Iceberg-defined S3 settings still fall back |
| partition spec | any, except an identity partition on a `float` or `double` column, or a `void` field whose source column was dropped beside a live field (see below) |
| partition spec | any, except a `void` field whose source column was dropped beside a live field (see below) |
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) and the v3 types `variant`, `unknown`, `timestamp_ns`, `geometry` and `geography` |

Within the namespaces that shape data-file bytes — `write.parquet.*` and `parquet.*` —
Expand Down Expand Up @@ -254,13 +254,6 @@ native write. If Iceberg's AWS property classes cannot be loaded, vendor `s3.*`
fall back too and planning still completes. The fall-back reason reports only sorted property
names, never their values, so credentials and tokens do not enter EXPLAIN or plan logs.

An identity partition on a `float` or `double` column falls back. iceberg-rust compares float
partition values with an equality that treats `-0.0` and `0.0` as one value, so the native writer
would put rows with either value in the same partition, where iceberg-java writes two. A read
that prunes on the other value's partition would then miss rows. The fall-back stays until
iceberg-rust distinguishes the two values
([apache/iceberg-rust#3325](https://github.com/apache/iceberg-rust/issues/3325)).

On a format-version 3 table with Iceberg 1.10 or newer, a write that rewrites existing rows falls
back. Copy-on-write `DELETE`, `UPDATE` and `MERGE` and `rewrite_data_files` write the row lineage
columns `_row_id` and `_last_updated_sequence_number` into the new data files, so that rewritten
Expand Down Expand Up @@ -391,9 +384,7 @@ a data file but not what any reader computes from it:
(iceberg-java names files `<partition>-<task>-<operation>-<count>`; iceberg-rust uses a
process-local counter).
- Partition directory names match iceberg-java 1.8+'s `PartitionSpec.partitionToPath` for every
partition type the native writer accepts. Identity partitions on `float` and `double` columns
fall back (see [Native Parquet write eligibility](#native-parquet-write-eligibility)), so
iceberg-java names those directories itself. On Iceberg 1.5.x,
partition type the native writer accepts. On Iceberg 1.5.x,
which the Spark 3.4 profile pins, iceberg-java itself spelled `timestamp` and `timestamptz`
directories with `LocalDateTime.toString()` / `OffsetDateTime.toString()`
(`ts=1969-12-31T23:59:58.500Z`) and left the partition field name unescaped; Comet uses the
Expand Down
114 changes: 98 additions & 16 deletions native/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions native/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ aws-sdk-sts = { version = "1.114.0", default-features = false }
# Pinned to a revision rather than a release because no published iceberg-rust
# version is on our arrow: 0.10.1 requires arrow/parquet ^58. Move to a plain
# version requirement once a release ships on arrow 59. Bump policy: #5645.
iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "af1da4c5bc86178c38c1c6db0578bdd9fb7021e3" }
iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust", rev = "af1da4c5bc86178c38c1c6db0578bdd9fb7021e3", features = ["opendal-memory", "opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b" }
iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust", rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b", features = ["opendal-memory", "opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
reqsign-core = "3"

[profile.release]
Expand Down
13 changes: 5 additions & 8 deletions native/core/src/execution/operators/iceberg_partition_value.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ use iceberg::{Error, ErrorKind, Result};
/// `timestamp`, or `timestamptz` source, which go through Comet's `iceberg_years` /
/// `iceberg_months` / `iceberg_days` / `iceberg_hours` kernels instead: the ones the sort in front
/// of a clustered write runs, pinned against iceberg-java's `DateTimeUtil` over the whole domain.
/// iceberg-rust's transforms differ from iceberg-java's in three ways:
/// iceberg-rust's transforms differ from iceberg-java's in two ways:
///
/// - `year` and `month` split the calendar with Arrow's `date_part`, which returns NULL for
/// anything `chrono` cannot represent -- past year 262142 -- whereas iceberg-java goes through
Expand All @@ -49,9 +49,6 @@ use iceberg::{Error, ErrorKind, Result};
/// - All four floor a pre-epoch timestamp that lies exactly 999999 microseconds into a unit, which
/// iceberg-java puts in the unit before, so `1969-01-01T00:00:00.999999` belongs in the 1968
/// partitions (apache/datafusion-comet#6426).
/// - `day` moves a timestamp from the last second of a day before 1969-12-31 into the next day,
/// unless its microsecond of second is 0 or 999999: it takes the whole seconds with a truncating
/// division and the microseconds with a flooring one (apache/iceberg-rust#3315).
///
/// A partition value that differs from the sort key can also fail a clustered write, which rejects
/// a row whose partition it has already closed. Everywhere else the two implementations agree, so
Expand Down Expand Up @@ -393,7 +390,8 @@ mod tests {
Some(-31_535_999_000_001),
// 1969-12-31T23:00:00.999999, where that moves only the hour.
Some(-3_599_000_001),
// 1969-12-30T23:59:59.5 and 1969-12-30T23:59:59.999998.
// 1969-12-30T23:59:59.5 and 1969-12-30T23:59:59.999998, which iceberg-rust's `day`
// moved into 1969-12-31 until apache/iceberg-rust#3315 was fixed.
Some(-86_400_500_000),
Some(-86_400_000_002),
None,
Expand All @@ -419,7 +417,7 @@ mod tests {
(
Transform::Day,
day([-366, -1, -2, -2]),
day([-365, -1, -1, -1]),
day([-365, -1, -2, -2]),
),
(
Transform::Hour,
Expand All @@ -438,8 +436,7 @@ mod tests {
for (i, (transform, java, rust)) in cases.iter().enumerate() {
let label = format!("{transform} of {}", source.data_type());
assert_eq!(&comet[i], java, "{label}");
// The reason Comet computes these: iceberg-rust floors the first two rows, and its
// `day` moves the last two into 1969-12-31 (apache/iceberg-rust#3315). If this
// The reason Comet computes these: iceberg-rust floors the first two rows. If this
// starts failing, iceberg-rust's transforms have changed and delegating needs
// another look.
assert_eq!(&iceberg_rust[i], rust, "iceberg-rust's {label}");
Expand Down
Loading
Loading