Core: Make partition field in TrackedFile optional by gaborkaszab · Pull Request #17000 · apache/iceberg

gaborkaszab · 2026-06-29T08:46:57Z

No description provided.

stevenzwu · 2026-06-29T21:38:37Z

 /** Adapts {@link TrackedFile} entries to the {@link DataFile} and {@link DeleteFile} APIs. */
 class TrackedFileAdapters {

+  static final Types.StructType EMPTY_STRUCT_TYPE = Types.StructType.of();


we can probably use the package private constants from the BaseFile class.

You're right. I removed introducing it here and reuse the on in BaseFile

ebyhr · 2026-06-29T22:07:43Z

    @Override
    public StructLike partition() {
-      return file().partition();
+      return file().partition() != null ? file().partition() : EMPTY_PARTITION_DATA;


nit: We could use the MoreObjects.firstNonNull method. Feel free to disregard this comment since this method isn't widely used in the codebase.

I just did a grep on the Java codebase, apparently, there is a single usage of this. I'd prefer to keep the ternary as is to follow the pattern around these files.

gaborkaszab

Thanks for the reviews, @stevenzwu , @ebyhr !

gaborkaszab · 2026-06-30T07:11:52Z

+  // partition is rebuilt with the supplied struct types inside schemaWithContentStats, so its
+  // ordinal is looked up by field ID.
  private static final int PARTITION_ORDINAL = ordinalOf(TrackedFile.PARTITION_ID);
-  private static final int CONTENT_STATS_ORDINAL = ordinalOf(TrackedFile.CONTENT_STATS_ID);


Note, this was unused, removed as part of this PR

Please revert unnecessary changes.

Found an overkill to open a new PR to remove an unused variable from the tests and rewrite/remove the relevant comment.

Reverted these now, opened a separate PR: #17031

gaborkaszab · 2026-06-30T07:12:20Z

 /** Adapts {@link TrackedFile} entries to the {@link DataFile} and {@link DeleteFile} APIs. */
 class TrackedFileAdapters {

+  static final Types.StructType EMPTY_STRUCT_TYPE = Types.StructType.of();


You're right. I removed introducing it here and reuse the on in BaseFile

dramaticlly · 2026-06-30T16:59:03Z

-    // should return EMPTY_PARTITION_DATA
-    assertThat(file.partition()).isNotNull();
-    assertThat(file.partition().size()).isEqualTo(0);
+    assertThat(file.partition()).isNull();


curious and non-blocking, if the intention is projection without partition. Do we need a coverage for projection where we have a valid/nonnull partition but not included in the projection as well?

This is okay, but callers should not access fields that have not been projected and we make no guarantees about it when they do. In many cases, we fail if possible like when accessing DataFile.recordCount() without projecting, which results in a NullPointerException because recordCount() returns a primitive long.

rdblue · 2026-06-30T23:12:08Z

    @Override
    public StructLike partition() {
-      return file().partition();
+      return file().partition() != null ? file().partition() : BaseFile.EMPTY_PARTITION_DATA;


Instead of relying on BaseFile, I'd prefer to move this constant to PartitionData.empty() or similar.

I introduced PartitionData.EMPTY for this

rdblue · 2026-06-30T23:14:31Z

-  // partition and content_stats are rebuilt with the supplied struct types inside
-  // schemaWithContentStats, so their ordinals are looked up by field ID.
+  // partition is rebuilt with the supplied struct types inside schemaWithContentStats, so its
+  // ordinal is looked up by field ID.


I would rather not have comments than have these that get updated every time an LLM touches the file.

seem my comment below

rdblue

Overall this looks okay. The main thing is that we should not use BaseFile.EMPTY_PARTITION_DATA without moving it to a better place. Tests look fine for this update.

gaborkaszab

Thanks for taking a look, @rdblue !

gaborkaszab · 2026-07-01T12:53:22Z

    @Override
    public StructLike partition() {
-      return file().partition();
+      return file().partition() != null ? file().partition() : BaseFile.EMPTY_PARTITION_DATA;


I introduced PartitionData.EMPTY for this

gaborkaszab · 2026-07-01T12:55:08Z

-  // partition and content_stats are rebuilt with the supplied struct types inside
-  // schemaWithContentStats, so their ordinals are looked up by field ID.
+  // partition is rebuilt with the supplied struct types inside schemaWithContentStats, so its
+  // ordinal is looked up by field ID.


seem my comment below

gaborkaszab · 2026-07-01T13:12:47Z

+  // partition is rebuilt with the supplied struct types inside schemaWithContentStats, so its
+  // ordinal is looked up by field ID.
  private static final int PARTITION_ORDINAL = ordinalOf(TrackedFile.PARTITION_ID);
-  private static final int CONTENT_STATS_ORDINAL = ordinalOf(TrackedFile.CONTENT_STATS_ID);


Found an overkill to open a new PR to remove an unused variable from the tests and rewrite/remove the relevant comment.

Reverted these now, opened a separate PR: #17031

github-actions Bot added the core label Jun 29, 2026

gaborkaszab mentioned this pull request Jun 29, 2026

Core: Adjustments for TrackedFileBuilder #16964

Open

wombatu-kun reviewed Jun 29, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/TrackedFile.java

gaborkaszab force-pushed the main_trackedfile_partition_optional branch from d6cf350 to ebe1087 Compare June 29, 2026 14:08

wombatu-kun approved these changes Jun 29, 2026

View reviewed changes

stevenzwu added this to V4: metadata tree Jun 29, 2026

stevenzwu moved this to In review in V4: metadata tree Jun 29, 2026

stevenzwu approved these changes Jun 29, 2026

View reviewed changes

ebyhr approved these changes Jun 29, 2026

View reviewed changes

gaborkaszab force-pushed the main_trackedfile_partition_optional branch from ebe1087 to 21f8c80 Compare June 30, 2026 07:13

gaborkaszab commented Jun 30, 2026

View reviewed changes

gaborkaszab mentioned this pull request Jun 30, 2026

Core: Add v4 manifest reader #16958

Open

dramaticlly approved these changes Jun 30, 2026

View reviewed changes

rdblue reviewed Jun 30, 2026

View reviewed changes

rdblue requested changes Jun 30, 2026

View reviewed changes

Core: Make partition field in TrackedFile optional

4ed590c

gaborkaszab force-pushed the main_trackedfile_partition_optional branch from 21f8c80 to 4ed590c Compare July 1, 2026 13:15

gaborkaszab commented Jul 1, 2026

View reviewed changes

gaborkaszab requested a review from rdblue July 1, 2026 13:16

anoopj approved these changes Jul 1, 2026

View reviewed changes

Uh oh!

Conversation

gaborkaszab commented Jun 29, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants