ORC: fill initial defaults for missing id-bound fields

[cbb330]

Summary

Fills a field's initial-default when it is absent from an id-bearing ORC file, using the same fallback order as the Parquet reader.

• buildOrcProjection omits an absent defaulted field only when the file has embedded IDs and the configured reader explicitly declares supportsInitialDefaults().
• The id-binding StructReader from ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #265 then materializes the declared default as a per-file constant, consuming no column vector.
• A present column always wins, including explicit null. ID-less/name-mapped files keep legacy null synthesis.
• Readers that do not opt in retain their previous projection shape and behavior.
• Spark 3.1 scans projecting a default at any nesting level are routed away from vectorized ORC reads; default materialization remains iterative-only.

Why reader opt-in is required

OrcIterable is shared by Generic, Spark, and Flink row and vectorized readers. Embedded field IDs establish that an absent column was never written, but do not establish that the selected reader can fill it. The new ORC.ReadBuilder.supportsInitialDefaults() capability keeps omission local to readers that implement the missing-field contract.

Generic row reads opt in here. Spark 3.1 row reads opt in in #264. This branch also provides the recursive Spark 3.1 scan-routing guard required before the row-reader implementation is enabled. Other engines and vectorized readers do not opt in.

Provenance

This is net new rather than a backport. Apache Iceberg 1.2 has the schema model for defaults but does not implement ORC default reads. The resolution logic mirrors Parquet's field-reader fallback: physical reader → declared default → null/failure.

Supported scope

Generic ORC row reader; scalar defaults at any struct nesting level, including structs inside list elements and map values.

Vectorized ORC default materialization remains deferred. Predicates on omitted defaulted fields use a conservative ORC SearchArgument so they are evaluated after iterative default materialization.

Stack

1. ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #265 — ID-based StructReader backport
2. This PR — Generic ORC initial-default fill, reader capability gate, and Spark 3.1 iterative-routing guard
3. Spark 3.1: fill ORC initial defaults on the row reader #264 — Spark 3.1 ORC initial-default fill

Testing Done

• Projection tests cover opted-in embedded-ID omission and non-opted null synthesis
• Generic ORC end-to-end coverage for scalar, required, nested, list, and map defaults
• Present values and explicit nulls override defaults
• ID-less files retain null behavior
• Top-level and nested projected defaults disable Spark 3.1 vectorized ORC reads
• Schemas without projected defaults remain vectorization eligible
• Full iceberg-orc and iceberg-data test suites pass
• Spotless passes

[mkuchenbecker]

what are trusted IDs?

[cbb330]

name is a bit vague so I changed it to a two-valued enum, which names where the field ids in the schema came from.

• EMBEDDED: the ids were read from iceberg.id column attributes persisted in the ORC file. When it is present it is great, a field with no id was genuinely never written, so its declared default can be filled safely.
• NAME_MAPPED: the file carried no ids of its own (e.g. was renamed from kafka ETL's Hive-migration), so an earlier step derived them at read time by matching column names. When file carries no ids, "absent" is not proof since e.g. an iceberg column can rename and the name mapping no longer covers.but it's data is physically in the file. Filling a default there would fabricate values over real data, so we don't fill the default on this path. We keep the original behavior here, in order to focus only on the important case: iceberg created files.

[mkuchenbecker]

actual logic

[mkuchenbecker]

overall makes sense, datatypes tested, where there is a default and the default is not specified, then populate.

An intergration test in li-openhouse for trino and spark will be useful to capture actual behaviour.

[cbb330]

I added an explicit supportsInitialDefaults() flag to the ORC row-reader path.

Previously, the shared ORC projection omitted missing defaulted columns whenever a file had embedded field IDs. That inadvertently affected every Spark/Flink row and vectorized reader, even though only Generic and Spark 3.1 iterative readers know how to fill the omitted column today. This could break non-supported engines by giving their positional readers a shorter column list.

Now a field is omitted only when all four conditions hold:

  supportsInitialDefaults
      && field.initialDefault() != null
      && !mapping.containsKey(field.fieldId())
      && fieldIdSource == FieldIdSource.EMBEDDED

Generic and Spark 3.1 iterative readers explicitly opt in. Other engines retain the previous projection and behavior. Spark 3.1 also avoids vectorized ORC reads when the projected schema contains
initial defaults; vectorized support can be added separately.

This keeps default filling isolated to readers that implement it, while preserving backward compatibility everywhere else.