Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790

Describe the bug

On an Iceberg table whose data files have no Iceberg field ids and are read through a name mapping (schema.name-mapping.default, which the migrate, snapshot and add_files procedures set), the native Iceberg scan returns NULL for nested fields whenever the query also reads a top-level primitive column. That covers struct fields, list element fields and map value fields. Spark returns the values, and nothing fails, so the wrong results are silent.

iceberg-rust applies the mapping in apply_name_mapping_to_arrow_schema (crates/iceberg/src/arrow/reader/projection.rs), which assigns field ids to the file's top-level Arrow fields only. Nested fields get no id, so the field-id projection never selects their leaves. When the query also reads a top-level primitive, the projection reads only that column, and RecordBatchTransformer fills the struct, list or map column with NULL as though the file didn't have it. When the query reads only nested fields, no leaf matches at all, the reader falls back to reading every column, and the values come out right. Iceberg Java's ApplyNameMapping visitor applies the mapping at every level of the schema.

The iceberg-rust code is the same at Comet's current pin, at bb1e4a4 (the 1.1 pin), and on iceberg-rust's main, so the 1.1.0 release has the bug too. apache/iceberg-rust#1845 tracks nested name mapping there as a missing feature.

Steps to reproduce

// Parquet files written without Iceberg field ids
spark.sql("""SELECT CAST(id AS INT) AS id,
    named_struct('a', CAST(id AS INT), 'b', CAST(-id AS INT)) AS s,
    array(named_struct('x', CAST(id AS INT))) AS items,
    map('k', named_struct('v', CAST(id AS INT))) AS m
  FROM range(50)""").coalesce(1).write.parquet(dataPath)

spark.sql("""CREATE TABLE cat.db.t (id INT, s STRUCT<a: INT, b: INT>,
  items ARRAY<STRUCT<x: INT>>, m MAP<STRING, STRUCT<v: INT>>) USING iceberg""")
spark.sql(s"CREATE TABLE src USING parquet LOCATION '$dataPath'")
SparkTableUtil.importSparkTable(spark, TableIdentifier("src"), table, stagingDir)

// importSparkTable sets no mapping. migrate, snapshot and add_files set this one.
table.updateProperties()
  .set("schema.name-mapping.default", NameMappingParser.toJson(MappingUtil.create(table.schema())))
  .commit()
spark.sql("REFRESH TABLE cat.db.t")

On main at 7d29453 (Spark 4.1, Iceberg 1.11):

Spark 3.4 with Iceberg 1.5.2 gives the same NULLs.

Expected behavior

The native scan returns the same values as Spark. The fix belongs in iceberg-rust, where apply_name_mapping_to_arrow_schema would apply the mapping to nested fields as Iceberg Java does. Until Comet's pin has that, CometScanRule could fall back when the table has a name mapping and the scan reads a struct, list or map column. Comet can't tell at planning time which data files lack field ids, so that guard would also catch tables whose files were all rewritten with ids after the migration.

Additional context

Found while reviewing #6725, which doesn't change it: the results are the same with its nested schema pruning on or off. Comet's existing migrated-table tests use SparkTableUtil.importSparkTable, which sets no name mapping, so no test reads a nested column through one.