[OSSIE][DATABRICKS] Add Databricks Unity Catalog Metric View converter

[Haoranli503]

Add Databricks Unity Catalog Metric View converter

Adds a bidirectional, offline converter between Apache Ossie semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the Databricks spoke in the hub-and-spoke architecture (DATABRICKS is already a listed vendor in converters/index.md). Validated against the live Databricks Metric View engine.

What's included

• Export (ossie-databricks export): Apache Ossie semantic model -> Metric View (one fact source + a nested joins tree, flattened dimensions, measures).
• Import (ossie-databricks import): Metric View -> Apache Ossie. Metric-View-only features (filter, window, format, rely, cardinality, parameters, materialization) are preserved in custom_extensions[DATABRICKS], so MV -> Apache Ossie -> MV is lossless.
• A unified ossie-databricks CLI and a string-in / string-out Python API.
• A README with a mapping table and limitations.

Design highlights

• Round-trip fidelity via the custom_extensions[DATABRICKS] stash -- the approach the converter guide recommends.
• Join tree, not flat joins. The Apache Ossie relationship graph is reassembled into the Metric View's nested fact + joins tree; a dataset reached by multiple paths (a diamond) is fanned out into one aliased join per path; cyclic graphs are rejected.
• Cardinality from direction. many_to_one / one_to_many is derived from the relationship from/to direction relative to a selectable grain (--source). Multiple facts are supported only narrowly -- when they share a conformed dimension (named as the source); a general galaxy schema (facts each with their own dimensions) or facts sharing no dimension can't map to one Metric View and are rejected with a clear error, never silently mis-converted.
• Key bridge. A primary_key/unique_keys whose columns cover a join key becomes rely: {at_most_one_match: true}, and vice versa.
• On export, joined-table columns are alias-qualified in dimension/measure expressions; fact columns in measures are emitted bare (the DBR idiom).
• Fails loudly, never silently. A join condition an Apache Ossie relationship can't represent (a non-equi or cross join), a cyclic or ambiguous-grain graph, and an unsupported version are rejected with a clear ConversionError; anything dropped (a foreign-vendor extension, an Apache-Ossie-only annotation, a non-DATABRICKS/ANSI_SQL dialect) emits a warning naming the object.

Conventions followed

• Lives at converters/databricks/, packaged like the dbt/gooddata spokes: pyproject.toml (hatchling, Apache-2.0), src/ossie_databricks/, tests/.
• Every source file carries the ASF license header, matching the sibling spokes.
• On export, dialect selection prefers DATABRICKS, falls back to ANSI_SQL.
• The only runtime dependency is PyYAML; Python 3.11+.

Testing

cd converters/databricks
pip install -e . && pip install pytest hypothesis
python -m pytest tests/

Example-based unit tests plus Hypothesis property-based round-trip tests in both directions, with a normalized TPC-DS fixture as a baseline.

Notes

• Targets Databricks Unity Catalog Metric View YAML v1.1.
• Foreign-vendor custom_extensions have no slot in the Metric View YAML; they are dropped on export with a warning.

Credits

@jackstein21 is credited as a co-contributor (see the commit's Co-authored-by trailer), including the property-based round-trip testing approach.

Related Issues

Will close #178, Ossie <-> Databricks converters will be tracked here.

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue
• Breaking changes to the spec are clearly called out in the summary

Ontology

• Ontology changes in ontology/ are consistent with spec changes
• New or modified terms are defined and documented

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes
• New converters include tests under the converter's test directory

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate
• CONTRIBUTING.md is updated if the contribution process changed

Examples

• examples/ are added or updated for any new spec constructs or converter support

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval

[Haoranli503]

RE @jbonofre 's review on #178:
1, Renamed OSI to Ossie, done. Two "OSI"s in the ossie converter I left as-is on purpose:

$schema=.../core-spec/osi-schema.json — points at a real file in upstream apache/ossie (never renamed; all sibling spokes use this exact name). Renaming it breaks the schema link.

"License :: OSI Approved :: Apache Software License" — this OSI = Open Source Initiative, not Ossie. Fixed PyPI classifier, only valid spelling for Apache-2.0; editing it fails packaging.

Renaming either would be a bug, not a cleanup.

2, Duplicate dimension/measure name is now a ConversionError, done. Previously a name collision (a dimension/measure sharing a name, case-insensitively) only emitted a warning and still wrote the duplicate to the output. It now raises a ConversionError naming the offending field, consistent with the rest of the converter — the user is told exactly what to rename before handing a broken file to Databricks. Added tests for both the duplicate-dimension and measure-collides-with-dimension cases.

[jbonofre]

I think the json.JSONDecodeError leak is worth to fix.

[jbonofre]

If someone edits (by hand) an Apache Ossie file and produces invalid JSON in a custom_extensions[DATABRICKS].data field, this raises a raw json.JSONDecodeError traceback rather than a ConversionError.

Maybe worth to wrap like this:

Suggested change

	data = json.loads(ext.get("data") or "{}")
	try:
	data = json.loads(ext.get("data") or "{}")
	except json.JSONDecodeError as e:
	raise ConversionError(f"DATABRICKS custom_extensions data is not valid JSON: {e}") from e

[Haoranli503]

Ah good catch, fixed!

[jbonofre]

The version check in convert_ossie_to_metric_view is exact match, so this converter will silently reject every Apache Ossie file the moment the spec version moves.

The dbt converter avoids this by depending on the apache-ossie package. This converter intentionally has no such dependency (that's ok), but there is no comment flagging that this constant must be updated in lockstep with core-spec.

I would suggest a comment to "track" this.

[Haoranli503]

Added a comment

[jbonofre]

write_stash always injects "_v": 1. The equivalent fixtureB_ossie.yaml has "_v": 1, ... in all its blobs. The missing marker means tpcds_ossie.yaml is inconsistent with what the converter actually writes. The canon() helper in _util.py parses JSON to dicts (masking the discrepancy), but a contributor comparing hand-authored fixture output to real converter output will see a difference.

[Haoranli503]

right, we need to be aligned, fixed!

[jbonofre]

That's a great contribution! Thanks!