Skip to content

Add Apache Polaris converter module - #94

Merged
jbonofre merged 1 commit into
mainfrom
polaris-converter
May 12, 2026
Merged

Add Apache Polaris converter module#94
jbonofre merged 1 commit into
mainfrom
polaris-converter

Conversation

@jbonofre

Copy link
Copy Markdown
Member

Summary

  • Adds a two-way converter between OSI semantic models and Apache Polaris catalogs under converters/polaris/
  • Import mode: reads Polaris namespaces/tables via the Iceberg REST Catalog API and generates OSI YAML, mapping Iceberg schemas to OSI datasets, fields, primary keys, and temporal dimensions
  • Export mode: reads OSI YAML and creates Polaris namespaces and Iceberg tables with type inference from field names and dimension metadata

Test plan

  • Unit tests pass (9 tests covering parsing, YAML generation round-trip, Iceberg schema generation, type inference, and edge cases)
  • Integration test against a running Polaris instance (import and export)
  • Verify round-trip fidelity: Polaris → OSI → Polaris

Introduces a Java/Maven converter under converters/polaris/ that provides
two-way conversion between OSI semantic models and Apache Polaris catalogs
via the Iceberg REST Catalog API.

Import mode reads Polaris namespaces and tables, mapping Iceberg schemas
to OSI datasets and fields. Export mode creates Polaris namespaces and
Iceberg tables from OSI semantic models with type inference.
@jbonofre

Copy link
Copy Markdown
Member Author

This is a use example:

  # Import: Polaris catalog → OSI YAML
  java -jar osi-polaris-converter.jar import \
    --url http://localhost:8181 --catalog my_catalog \                                                                                                                                                                                        
    --client-id <id> --client-secret <secret> \
    -o output.yaml                                                                                                                                                                                                                            
                                                            
  # Export: OSI YAML → Polaris catalog                                                                                                                                                                                                        
  java -jar osi-polaris-converter.jar export \              
    --url http://localhost:8181 --catalog my_catalog \
    --token <bearer-token> \                                                                                                                                                                                                                  
    model.yaml

I implemented with the following logic:

  • Each Polaris namespace maps to an OSI semantic model
  • Each Iceberg table maps to an OSI dataset
  • Iceberg schema fields map to OSI fields with ANSI_SQL expressions
  • Iceberg identifier-field-ids map to OSI primary_key
  • Temporal Iceberg types (timestamp, timestamptz, date) set dimension.is_time: true
  • Table properties are preserved in custom_extensions

@jbonofre
jbonofre requested a review from khush-bhatia March 23, 2026 12:43
/**
* Java representation of an OSI semantic model parsed from YAML.
*/
public class OsiModel {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this, @jbonofre. Can we put it a common module instead of Polaris module? I think the Java representation would be really useful for other catalogs as well.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flyrain That's a good idea. I think it also related to a discussion we had before: what should be the "language layout" as we have converter in Python, Java, ...

Let me give a proposal.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Looking forward to it

@jbonofre jbonofre Apr 21, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working on a PR to have core-spec module, that will create bindings for different languages (including java).

@jbonofre

jbonofre commented Apr 21, 2026

Copy link
Copy Markdown
Member Author

@khush-bhatia I'm preparing a first step PR with core-spec and java binding.

@sgandon sgandon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jbonofre

Copy link
Copy Markdown
Member Author

I created #121 for follow up.

@jbonofre
jbonofre merged commit 36b9f65 into main May 12, 2026
@jbonofre
jbonofre deleted the polaris-converter branch May 12, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants