Losing "element-id" after calling BuildAvroProjection

[haizhou-zhao]

Hello,

We are using iceberg 0.11.x, and we had an issue with assembling an iceberg schema from the result of AvroSchemaUtil.buildAvroProjection.

First, couple words on the overall architecture of our operation: we manage iceberg tables for our customers and accept schema evolution requests from them in the format of Avro. This is our schema compatibility check:

Given: [avro.Schema] currentSchema, [avro.Schema] updatedSchema

step 1: currentIcebergSchema = AvroSchemaUtil.toIceberg(currentSchema)

step 2: projectedIds = TypeUtil.getProjectedIds(currentIcebergSchema)

step 3: nameMapping = MappingUtil.create(currentIcebergSchema)

step 4: prunedSchema = AvroSchemaUtil.pruneColumns(updatedSchema, projectedIds, nameMapping)

step 5: projectionSchema = AvroSchemaUtil.buildAvroProjection(prunedSchema, currentIcebergSchema, HashMap.empty())

step 6: updatedIcebergSchema = AvroSchemaUtil.toIceberg(projectionSchema)

step 7: CheckCompatibility.writeCompatibilityErrors(currentIcebergSchema, updatedIcebergSchema)

Recently, one of our customer submitted a schema change where they want to add more optional fields to the element type of an array column.

<excerpt of schema change>
{
    "type": "record",
    "name": "top-level",
    "fields": [
         {
               "type": "array",
               "items":  {
                       "type": "record",
                       "name": "element-record",
                       "fields": [
                           {
                                  "name": "field1",
                                  "type": ["null", "int"],
                           },
+                           {
+                                  "name": "field2",
+                                  "type": ["null", "string"],
+                           },
                       ]
                }
         }
    ]
}

Then our compatibility check breaks at step 6 (java.lang.IllegalArgumentException: Multiple entries with same key:). What we've discovered so far are:

1. each of the Avro node is assigned an id after step 5 for prunedSchema
2. the "top-level.element" node lost its assigned id from the prunedSchema after step 6 in projectionSchema reference
3. when reassembling an Iceberg schema out of projectionSchema at step 7, the "top-level.element" node, as it does not have "element-id" property, is assigned a generated id that happens to collide its id with another node reference

The questions are:

1. When projecting schema of array type, even if the underlying schema differs, should the "element-id" property be preserved? Reference
2. If not, then is there an Iceberg API/Util that performs projection (column reordering, etc.) while preserving ids of all node from the projected schema?

[RussellSpitzer]

Have you tried this in the most recent release (0.13) ? There have been several pruning fixes.

#2952
#2877

[haizhou-zhao]

Hello Russell,

Thanks for your reply. We can definitely try out the pruning fixes you mentioned. However, we lost the id information after we call AvroSchemaUtil.buildAvroProjection, not the pruning. Specifically, we lost the list level id when we extract element schema but leave out all the list level properties. Reference. Is there anyway we could handle that?

[haizhou-zhao]

To summarize, the issue is incurred because ID properties are not preserved when projecting list schema. (We also found the same issue for map schema). It should be fixed by #4120.

Thanks @RussellSpitzer for helping out with the issue. I'm closing this as the PR mentioned above is fixed.

Also thanks to @rdblue @szehon-ho for reviewing the PR.

[szehon-ho]

Nice work!