diff --git a/ROADMAP.md b/ROADMAP.md index 5ca262c7..1fad2ba6 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -434,6 +434,7 @@ Broad ecosystem adoption depends on practical tools that let teams validate thei - [GoodData Converter](converters/gooddata/) — bidirectional Ossie ↔ GoodData LDM converter - [Salesforce Converter](converters/salesforce/) — Ossie ↔ Salesforce converter - [Apache Polaris Converter](converters/polaris/) — Ossie → Apache Polaris converter +- [OrionBelt Converter](converters/orionbelt/) — bidirectional Ossie ↔ OrionBelt OBML converter **Related Issues:** diff --git a/converters/orionbelt/README.md b/converters/orionbelt/README.md new file mode 100644 index 00000000..bcc96bb3 --- /dev/null +++ b/converters/orionbelt/README.md @@ -0,0 +1,125 @@ +# apache-ossie-orionbelt + +Bidirectional converter between **OBML** (OrionBelt Markup Language) semantic +models and **OSI** ([Open Semantic Interchange](https://open-semantic-interchange.org/)), +the open standard for portable semantic models (metrics, dimensions, +relationships). + +This package is licensed under **Apache-2.0** and may be used freely. It is the +OrionBelt converter in the OSI converter ecosystem. The canonical source is +developed in the +[orionbelt-semantic-layer](https://github.com/ralfbecher/orionbelt-semantic-layer) +repository (under `packages/osi-orionbelt`) and published to PyPI from there; +file issues and contributions upstream. + +## Requirements + +- Python 3.12+ +- [uv](https://docs.astral.sh/uv/) (recommended) or pip + +## Install + +```bash +pip install apache-ossie-orionbelt +``` + +Optional deep OBML semantic validation (cycles, duplicate names, invalid refs) +via the full OrionBelt engine: + +```bash +pip install "apache-ossie-orionbelt[obml-validation]" +``` + +Without that extra, OBML validation runs JSON-schema checks only and emits a +warning for the deeper semantic pass. + +## CLI + +A single `ossie-orionbelt` command with two subcommands (mirroring `ossie-dbt`): + +| Subcommand | Direction | In | Out | +|---------|-----------|----|----| +| `obml-to-osi` | OBML -> OSI core-spec | OBML YAML | OSI YAML | +| `obml-to-osi --ontology` | OBML -> OSI ontology | OBML YAML | OSI ontology YAML | +| `osi-to-obml` | OSI core-spec -> OBML | OSI YAML | OBML YAML | + +```bash +ossie-orionbelt obml-to-osi -i model.obml.yaml -o model.osi.yaml +ossie-orionbelt obml-to-osi --ontology -i model.obml.yaml -o model.ontology.yaml +ossie-orionbelt osi-to-obml -i model.osi.yaml -o model.obml.yaml +``` + +`-i/--input` and `-o/--output` are required. Each subcommand prints conversion +warnings and a validation summary to stderr, and exits non-zero when the +produced document fails schema validation (unless `--no-validate`). Run +`ossie-orionbelt --help` or `ossie-orionbelt obml-to-osi --help` for the full +option list. + +## Python API + +```python +import yaml +from ossie_orionbelt import OBMLtoOSI, OSItoOBML, validate_osi + +obml = yaml.safe_load(open("model.obml.yaml")) +osi = OBMLtoOSI(obml, "sales", "Sales model").convert() +result = validate_osi(osi) +assert result.valid + +obml_again = OSItoOBML(osi).convert() +``` + +## Vendor extensions + +OSI `custom_extensions` carry vendor-tagged payloads. This converter: + +- emits OrionBelt/OBML-proprietary data under the **`ORIONBELT`** vendor on OBML + to OSI (OBML-only filters, settings, owner, refresh, type info, etc.); +- stashes OSI-native fields that OBML can't represent (unique keys, field + labels, leftover `ai_context`) under the **`OSI`** vendor when going OSI to + OBML, restoring them to first-class OSI fields on the way back; +- **preserves third-party vendor extensions verbatim** (e.g. `SNOWFLAKE`, + `DBT`, `SALESFORCE`, `GOODDATA`) at the model, dataset, field, and + measure/metric levels, so a full OSI to OBML to OSI roundtrip keeps the + original vendor and data. OSI has no separate dimension entity, so an OBML + dimension's foreign extensions surface on its OSI field. + +Legacy `COMMON` / `OBSL` tags from earlier converter versions are still accepted +on read. + +## Limitations / unsupported constructs + +Some OBML constructs have no native OSI equivalent and are carried in vendor +`custom_extensions` (`obml_*` payloads) so they round-trip without loss back to +OBML, but are not interpreted by other OSI consumers: + +- **Many-to-many joins** - represented in OBML join cardinality; flagged on + export. +- **Named secondary join paths** - OBML's multiple join paths between the same + pair of objects are an OBML-specific topology feature. +- **Measures / metrics and column-level value concepts in the ontology layer** - + not represented in the OSI ontology export. +- **OSI metrics with no OBML representation** - a metric whose only expression is + in a non-SQL dialect (`MDX`, `TABLEAU`, `MAQL`), or whose SQL expression cannot + be decomposed into OBML measures/metrics, is **not** dropped: the original OSI + metric is preserved verbatim in a model-level `OSI`-vendor `custom_extension` + (`obml_unconverted_metrics`) and re-emitted on OBML to OSI, so the OSI to OBML + to OSI roundtrip stays lossless. A `LOSSY:` warning is raised for each such + metric because it is **not queryable through OBML**. SQL expressions in the + `ANSI_SQL`, `SNOWFLAKE`, and `DATABRICKS` dialects are all read on import. + +OSI v0.1.x inputs are accepted on read via a legacy normalization shim; output +targets OSI **v0.2.0.dev0**. + +See [`osi_obml_mapping_analysis.md`](./osi_obml_mapping_analysis.md) for the +full OBML <-> OSI core-spec mapping and +[`osi_obml_ontology_mapping_analysis.md`](./osi_obml_ontology_mapping_analysis.md) +for the ontology-layer mapping and its documented gaps. + +## Development + +```bash +uv sync # install +uv run pytest # run the test suite (includes a TPC-DS baseline) +uv run ruff check && uv run mypy src/ossie_orionbelt +``` diff --git a/converters/orionbelt/osi_obml_mapping_analysis.md b/converters/orionbelt/osi_obml_mapping_analysis.md new file mode 100644 index 00000000..d94fe09c --- /dev/null +++ b/converters/orionbelt/osi_obml_mapping_analysis.md @@ -0,0 +1,269 @@ +# OSI ↔ OBML Mapping Analysis + +> Bidirectional conversion between [Open Semantic Interchange (OSI)](https://github.com/open-semantic-interchange/OSI) v0.2.0.dev0 and [OrionBelt ML (OBML)](https://github.com/ralfbecher/orionbelt-semantic-layer) v1.0 semantic model formats. OSI v0.1.x inputs are still accepted on read via a legacy normalization shim; output targets v0.2.0.dev0. + +## 1. Structural Comparison + +| Aspect | OSI v0.2.0.dev0 | OBML v1.0 | +|---|---|---| +| **Top-level** | `semantic_model[]` (array of models) | Single model with `dataObjects`, `dimensions`, `measures`, `metrics` sections | +| **Tables / Entities** | `datasets[]` (flat array) | `dataObjects{}` (named dictionary) | +| **Column identifiers** | `fields[].name` (snake_case code) | `columns{}.code` (with display name as dict key) | +| **Expressions** | `expression.dialects[]` per field (multi-dialect) | Single SQL expression via `code` (single dialect) | +| **Joins / Relationships** | `relationships[]` (global, separate section) | `joins[]` (inline on each data object) | +| **Dimensions** | `field.dimension.is_time` (inline flag on fields) | `dimensions{}` (separate top-level section) | +| **Measures** | N/A (merged into metrics) | `measures{}` (explicit aggregation definitions) | +| **Metrics** | `metrics[]` with full SQL expressions | `metrics{}` with `{[Measure]}` references | +| **AI Context** | `ai_context` on every entity | `customExtensions` with `vendor: "OSI"` | +| **Extensibility** | `custom_extensions[]` per entity | `customExtensions[]` per entity | +| **Keys** | `primary_key`, `unique_keys` on datasets | N/A | +| **Secondary joins** | N/A | `secondary: true`, `pathName` | +| **Fan-out protection** | N/A | `allowFanOut`, `reduceToRelationDimensionality` | + +## 2. Key Differences + +### 2.1 Naming Convention + +- **OSI** uses snake_case codes everywhere (`name: "store_sales"`) +- **OBML** supports dual naming — a display name as the dictionary key and a `code` for the physical SQL reference + +During OSI → OBML conversion, field names are used directly as both the display name and code. During OBML → OSI conversion, the `code` value becomes the OSI field `name`. + +### 2.2 Relationship Placement + +- **OSI** defines relationships globally, referencing dataset names by string +- **OBML** defines joins inline on the "from" side data object + +The converter restructures between these two representations automatically, preserving column mappings and generating descriptive relationship names. + +### 2.3 Measures vs. Metrics + +This is the most fundamental structural difference between the two formats. + +- **OSI** has a single "metrics" concept with full SQL expressions (e.g., `SUM(store_sales.ss_ext_sales_price)`) +- **OBML** explicitly separates: + - **Measures**: Simple aggregations on columns (e.g., `SUM` of `ss_ext_sales_price`) + - **Metrics**: Derived calculations referencing measures via `{[Name]}` syntax (e.g., `{[total_sales]} / {[customer_count]}`) + +The converter handles this decomposition automatically: + +| OSI metric type | OBML mapping | +|---|---| +| `AGG(dataset.column)` | Direct measure | +| `AGG(DISTINCT dataset.column)` | Measure with `distinct: true` | +| `AGG(expr)` (e.g., `SUM(a.x * a.y)`) | Expression-based measure | +| Multi-aggregation expression | Auto-generated measures + metric formula | + +**Example** — OSI metric `customer_lifetime_value`: +```yaml +# OSI +expression: SUM(store_sales.ss_ext_sales_price) / COUNT(DISTINCT customer.c_customer_sk) +``` + +is decomposed into OBML: +```yaml +# OBML measures (auto-generated) +measures: + total_sales: + columns: + - dataObject: store_sales + column: ss_ext_sales_price + resultType: float + aggregation: sum + + _customer_c_customer_sk_count_distinct: + columns: + - dataObject: customer + column: c_customer_sk + resultType: float + aggregation: count + distinct: true + +# OBML metric (references the measures) +metrics: + customer_lifetime_value: + expression: "{[total_sales]} / {[_customer_c_customer_sk_count_distinct]}" +``` + +When a simple OSI metric (e.g., `SUM(store_sales.ss_ext_sales_price)`) is equivalent to an existing named measure, the converter deduplicates and reuses the named measure rather than creating a redundant auto-measure. + +### 2.4 AI Context Preservation + +OSI's `ai_context` (instructions, synonyms, examples) is preserved losslessly during conversion via OBML's `customExtensions` mechanism: + +```yaml +# OSI input +ai_context: + synonyms: + - "sales transactions" + - "store purchases" + +# OBML output (via customExtensions) +customExtensions: + - vendor: OSI + data: '{"synonyms": ["sales transactions", "store purchases"]}' +``` + +This applies at all levels: datasets → data objects, fields → columns, and model-level `ai_context`. + +During OBML → OSI conversion, the `customExtensions` with `vendor: "OSI"` are read back and restored as native `ai_context` on the OSI side. + +### 2.5 OBML-Specific Features (Not Representable in OSI) + +These OBML features have no direct OSI equivalent. Where possible, metadata is preserved in OSI `ai_context` or `custom_extensions` (with `vendor_name: "COMMON"` and `obml_`-prefixed keys) for lossless roundtrip: + +- Secondary joins with `pathName` (preserved in relationship `ai_context`) +- `allowFanOut` — preserved in metric `custom_extensions` (`obml_allow_fan_out`) +- Dynamic date filters (`dynamicDate`, `dynamicDateRange`) — not yet preserved +- `timeGrain` on dimensions — preserved in field `custom_extensions` (`obml_time_grain`) +- Dimension `format` — preserved in field `custom_extensions` (`obml_dimension_format`) +- Measure filters — preserved in metric `custom_extensions` (`obml_filters`) +- Measure `total` — preserved in metric `custom_extensions` (`obml_total`) +- Measure `format` — preserved in metric `custom_extensions` (`obml_format`) +- Measure `delimiter` — preserved in metric `custom_extensions` (`obml_delimiter`) +- Measure `withinGroup` — preserved in metric `custom_extensions` (`obml_within_group`) +- Metric `format` — preserved in metric `custom_extensions` (`obml_format`) +- Locale settings — not yet preserved +- `abstractType` (OBML type system) — preserved in field `custom_extensions` (`obml_abstract_type`) + +### 2.6 OSI-Specific Features and How They Map to OBML + +- **`primary_key`** — natively represented: OSI's dataset-level `primary_key` array maps to per-column `primaryKey: true` on OBML columns (`DataObjectColumn.primaryKey`), and back to the dataset array on export. +- **`unique_keys`** — no native OBML equivalent; round-trips via an `OSI`-vendor `customExtension` (`obml_unique_keys`). +- **Multi-dialect expressions** — on import the converter reads the first available SQL dialect in the order `ANSI_SQL`, `SNOWFLAKE`, `DATABRICKS`; non-SQL dialects (`MDX`, `TABLEAU`, `MAQL`) are not parsed. A metric with no SQL-parseable dialect, or an expression OBML cannot decompose, is preserved verbatim (`obml_unconverted_metrics`) with a `LOSSY:` warning rather than dropped. On export, OBML measures/metrics emit `ANSI_SQL`. +- **`ai_context`** — preserved losslessly via `customExtensions` (see Section 2.4). +- **`custom_extensions`** — mapped to OBML `customExtensions`. + +## 3. Conversion Strategies + +### 3.1 OSI → OBML + +1. Parse `source` string to extract `database`, `schema`, and `table` +2. Convert fields to columns with type inference (heuristic-based `abstractType`) +3. Restructure global relationships into inline joins on data objects +4. Decompose metric SQL expressions into OBML measures + metrics +5. Extract dimension-flagged fields into the top-level `dimensions` section (excluding FK/PK join keys) +6. Preserve `ai_context` losslessly via `customExtensions` (vendor: `"OSI"`) + +### 3.2 OBML → OSI + +1. Combine `database.schema.code` into the OSI `source` string +2. Convert columns to fields with `ANSI_SQL` dialect expressions +3. Extract inline joins into global relationships with generated names +4. Convert measures to OSI metrics with SQL expressions +5. Expand metric templates by substituting measure SQL into `{[Name]}` references +6. Map OBML dimension metadata into `field.dimension.is_time` flags +7. Preserve secondary join info in relationship `ai_context` +8. Store OBML-specific type info in `custom_extensions` with `vendor_name: "COMMON"` + +## 4. Validation + +The converter includes dual-layer validation for both formats, ensuring that converted output is structurally and semantically correct. + +### 4.1 OBML Validation + +1. **JSON Schema** — validates against `schema/obml-schema.json` (Draft 7) +2. **Semantic** — runs OrionBelt's `ReferenceResolver` + `SemanticValidator` (reference integrity, cycle detection, multipath detection, duplicate identifiers) + +### 4.2 OSI Validation + +1. **JSON Schema** — validates against the OSI core `osi-schema.json` (Draft 2020-12), resolved from the repo's `core-spec/` (no vendored copy) +2. **Unique names** — checks uniqueness of dataset, field, metric, and relationship names +3. **References** — verifies that relationship `from`/`to` reference existing datasets + +Validation runs automatically after each conversion. Use `--no-validate` to skip. + +## 5. Converter Usage + +### CLI + +A single `ossie-orionbelt` command with two subcommands is installed with the package: + +```bash +# OSI → OBML +ossie-orionbelt osi-to-obml -i tpcds_osi.yaml -o tpcds_as_obml.yaml + +# OBML → OSI +ossie-orionbelt obml-to-osi -i tpcds_as_obml.yaml -o tpcds_obml_as_osi.yaml \ + --model-name tpcds_retail_model \ + --description "TPC-DS retail semantic model" + +# OBML → OSI ontology document +ossie-orionbelt obml-to-osi --ontology -i tpcds_as_obml.yaml -o tpcds_ontology.yaml + +# Skip validation +ossie-orionbelt osi-to-obml -i input.yaml -o output.yaml --no-validate +``` + +### CLI Options + +| Subcommand / Option | Description | +|---|---| +| `osi-to-obml` | Convert OSI → OBML | +| `obml-to-osi` | Convert OBML → OSI | +| `--ontology` | (`obml-to-osi`) emit an OSI ontology document instead of core-spec | +| `-i`, `--input` | Input file (required) | +| `-o`, `--output` | Output file (required) | +| `--model-name` | Model name for OBML → OSI | +| `--description` | Model description for OBML → OSI | +| `--ai-instructions` | AI instructions for OBML → OSI | +| `--database` | Default database for OSI → OBML (default: `ANALYTICS`) | +| `--schema` | Default schema for OSI → OBML (default: `PUBLIC`) | +| `--no-validate` | Skip post-conversion validation | + +### Python API + +```python +from ossie_orionbelt import OSItoOBML, OBMLtoOSI, validate_obml, validate_osi + +# OSI → OBML +converter = OSItoOBML(osi_dict) +obml = converter.convert() +result = validate_obml(obml) +assert result.valid + +# OBML → OSI +converter = OBMLtoOSI(obml_dict, model_name="my_model") +osi = converter.convert() +result = validate_osi(osi) +assert result.valid +``` + +## 6. Example: TPC-DS Roundtrip + +The converter is validated against the official [TPC-DS example](https://github.com/open-semantic-interchange/OSI/blob/main/examples/tpcds_semantic_model.yaml) from the OSI repository. That file is vendored at `tests/fixtures/tpcds_semantic_model.yaml` and exercised by `tests/test_osi_tpcds_baseline.py`, which runs the OSI converters guide's [conceptual conversion flow](https://github.com/open-semantic-interchange/OSI/blob/main/converters/index.md#example-conceptual-conversion-flow) end to end: OSI to OBML to OSI, asserting validity at each step and that the example's `SALESFORCE` and `DBT` custom extensions survive the round-trip (step 7). + +### OSI → OBML + +The TPC-DS OSI model with 5 datasets, 4 relationships, and 5 metrics converts cleanly to OBML: + +- 5 data objects with inline joins +- 16 dimensions (FK/PK join keys excluded) +- 5 measures (3 direct + 2 auto-generated for metric decomposition) +- 2 metrics (composite expressions referencing measures) +- All `ai_context` synonyms preserved via `customExtensions` + +### OBML → OSI (Roundtrip) + +Converting the OBML output back to OSI produces a valid OSI model where: + +- All `ai_context` synonyms are restored from `customExtensions` +- Measures are re-expanded into SQL metric expressions +- Inline joins are extracted back into global relationships +- OBML type information is preserved in `custom_extensions` + +### Files + +| File | Description | +|---|---| +| `tests/fixtures/tpcds_osi.yaml` | Official TPC-DS OSI example (from OSI repo) | +| `tests/fixtures/tpcds_as_obml.yaml` | Converted OBML output | +| `core-spec/osi-schema.json` | OSI core JSON Schema (Draft 2020-12), resolved at runtime; not vendored | +| `src/ossie_orionbelt/converter.py` | Bidirectional converter with validation | + +## 7. Future Considerations + +- **MCP/API integration** — Expose OSI import/export as OrionBelt MCP tools or REST API endpoints +- **Multi-dialect support** — Preserve non-ANSI dialect expressions during roundtrip +- **Primary keys in OBML** — Add optional `primary_key` to data objects for richer metadata +- **Vendor enum** — Register `"ORIONBELT"` as an OSI vendor for OBML-specific extensions diff --git a/converters/orionbelt/osi_obml_ontology_mapping_analysis.md b/converters/orionbelt/osi_obml_ontology_mapping_analysis.md new file mode 100644 index 00000000..0764bf1b --- /dev/null +++ b/converters/orionbelt/osi_obml_ontology_mapping_analysis.md @@ -0,0 +1,82 @@ +# OBML → OSI Ontology Mapping Analysis + +This pins the rules used by `OBMLtoOSIOntology` to derive an **OSI ontology +document** (OSI ontology format, version `0.2.0.dev0`) from an OBML semantic +model. + +The OSI ontology is a **separate document** from the OSI core-spec semantic +model (different `$id`, different required root). It is produced alongside the +core export, never merged into it — OSI's own `validation/validate.py` validates +one document against one schema and reads only the first YAML document, so a +combined or multi-doc file is not portable. See the `include_ontology` flag on +the export endpoints. + +## Document shape produced + +```yaml +version: 0.2.0.dev0 +name: # required +description: # optional +ai_context: { instructions: ... } # optional, from ai_instructions / model +ontology: # required, minItems 1 + - concept: + name: # = OBML dataObject display name + type: EntityType + description: ... + relationships: # outgoing joins keyed by this entity + - name: _to_ + roles: [{ concept: }] # declaring concept (A) is the implicit first role + multiplicity: ManyToOne | OneToOne + verbalizes: ["{} relates to {}"] +ontology_mappings: + - name: _map + semantic_model: { ...full OSI core-spec model... } # reused from OBMLtoOSI + concept_mappings: + - concept: + object_mappings: [{ expression: "." }] + link_mappings: + - relationship: _to_ + object_mapping: { concept: , expression: "." } +``` + +## Mapping rules + +| OBML construct | OSI ontology target | Notes | +|----------------|---------------------|-------| +| `dataObject` | `EntityType` `Concept` (one per object) | name = display name (matches OSI dataset name for ref consistency) | +| `dataObject.description` / `.comment` | `concept.description` | first non-empty wins | +| `join` (A → B) | `Relationship` under A's component | declaring concept A is the implicit first role; B is an explicit `role` | +| `join.joinType` | `Relationship.multiplicity` | `many-to-one`→`ManyToOne`, `one-to-one`→`OneToOne` | +| `column.primaryKey` | entity `object_mappings[].expression` | `
.`; identifies the entity | +| `join.columnsFrom` (FK) | `link_mappings[].object_mapping.expression` | `.`; binds the relationship to its far role | +| whole core model | `ontology_mappings[].semantic_model` | embedded verbatim from `OBMLtoOSI.convert()` | + +`
` is the final identifier of the dataset `source` (e.g. `db.schema.t` → `t`), +falling back to the dataset name when `source` has no dotted physical table. + +## Gaps and warnings (emitted to `warnings`) + +| OBML construct | Handling | Reason | +|----------------|----------|--------| +| `joinType: many-to-many` | relationship **skipped** + warning | OSI `Multiplicity` enum is only `ManyToOne`/`OneToOne` | +| missing/unknown `joinType` | defaults to `ManyToOne` + warning | matches core-converter default | +| secondary / `pathName` joins | emitted as ordinary relationships + warning | OSI ontology has no named-alternate-path concept | +| composite primary/foreign keys | first column used + warning | `object_mapping.expression` is a single scalar SQL expression | +| measures / metrics | **not** in the ontology layer | live only in the embedded core `semantic_model`; ontology models entities/relationships | +| columns (non-key) as value concepts | not modeled (entities only) | keeps v1 valid and focused; ORM-style `ValueType` modeling deferred | +| `verbalizes` / `derived_by` / `requires` / `identify_by` | `verbalizes` emitted as a generated stub; others omitted | OBML has no native source for fact-based verbalization or derivation | + +## Validation + +`validate_osi_ontology()` runs semantic checks only (ossie ships no OSI ontology +schema, and the converter emits ontology documents but never consumes them): +unique concept names; relationship `roles` reference defined concepts; +`concept_mappings` reference defined concepts. + +## Stability note + +OSI ontology is `0.2.0.dev0` (pre-1.0). This exporter is the supported, +schema-validated direction. An ontology **importer** (OSI ontology → OBML) is +intentionally deferred until OSI drops the `dev` pre-release suffix, because the +gaps above make the reverse direction lossy. Import the OSI **core spec** +instead (already supported). diff --git a/converters/orionbelt/pyproject.toml b/converters/orionbelt/pyproject.toml new file mode 100644 index 00000000..d0396570 --- /dev/null +++ b/converters/orionbelt/pyproject.toml @@ -0,0 +1,72 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "apache-ossie-orionbelt" +version = "0.1.0" +description = "Bidirectional OBML <-> OSI (Open Semantic Interchange) converter for OrionBelt semantic models" +license = { text = "Apache-2.0" } +readme = "README.md" +authors = [{ name = "Ralf Becher", email = "info@orionbelt.ai" }] +keywords = ["osi", "open-semantic-interchange", "obml", "semantic-layer", "orionbelt", "converter"] +requires-python = ">=3.12" +dependencies = [ + "pyyaml>=6.0", + "jsonschema>=4.18", + "referencing>=0.30", +] + +[project.optional-dependencies] +# Deep OBML semantic validation (cycles, duplicate names, invalid refs) via the +# full OrionBelt engine. Optional: validate_obml degrades to JSON-schema-only +# checks with a warning when this is not installed. +obml-validation = ["orionbelt-semantic-layer"] +dev = [ + "pytest>=8.0", + "mypy>=1.10", + "ruff>=0.4", + "types-jsonschema", + "types-PyYAML", +] + +[project.scripts] +ossie-orionbelt = "ossie_orionbelt.cli:main" + +[tool.hatch.build.targets.wheel] +packages = ["src/ossie_orionbelt"] + +# The OSI core schema is not tracked as a copy in this package (it lives once at +# the repo's core-spec/). Bundle it into the built wheel from that single source +# so a pip-installed converter still validates OSI documents self-contained. +# In-tree (dev/test) runs resolve it from core-spec/ via validation._osi_schema_path. +[tool.hatch.build.targets.wheel.force-include] +"../../core-spec/osi-schema.json" = "ossie_orionbelt/schemas/osi-schema.json" + +# Only the OBML schema (OrionBelt's own format) is vendored under +# src/ossie_orionbelt/schemas/. OSI core documents are validated against the +# repo's core-spec/osi-schema.json, resolved at runtime by +# validation._osi_schema_path (no duplicate copy). The OSI ontology export is +# validated semantically only (ossie ships no ontology schema). + +[tool.ruff] +line-length = 100 +target-version = "py312" + +[tool.ruff.lint] +select = ["E", "F", "I", "N", "UP", "B", "A", "SIM"] + +[tool.mypy] +python_version = "3.12" +files = ["src/ossie_orionbelt"] + +# The OrionBelt engine is an optional runtime dependency: validate_obml uses it +# for semantic validation when present and degrades gracefully otherwise, so it +# is not installed in this standalone converter and has no published stubs. +[[tool.mypy.overrides]] +module = ["orionbelt.*"] +ignore_missing_imports = true + +[tool.pytest.ini_options] +testpaths = ["tests"] +pythonpath = ["src"] diff --git a/converters/orionbelt/src/ossie_orionbelt/__init__.py b/converters/orionbelt/src/ossie_orionbelt/__init__.py new file mode 100644 index 00000000..5dc24fa6 --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/__init__.py @@ -0,0 +1,41 @@ +"""ossie-orionbelt: bidirectional OBML <-> OSI converter. + +Converts between OrionBelt Markup Language (OBML) semantic models and Open +Semantic Interchange (OSI) models, in both directions, plus an OSI ontology +emitter. Validation helpers check OBML and OSI documents against their JSON +schemas. + +Public API: + OSItoOBML - convert an OSI model dict to OBML + OBMLtoOSI - convert an OBML model dict to OSI core-spec + OBMLtoOSIOntology - emit an OSI ontology document from an OBML model + validate_obml - validate an OBML model dict + validate_osi - validate an OSI model dict + validate_osi_ontology - validate an OSI ontology document dict + ValidationResult - structured validation result +""" + +from __future__ import annotations + +from ossie_orionbelt.converter import ( + OBMLtoOSI, + OBMLtoOSIOntology, + OSItoOBML, + ValidationResult, + validate_obml, + validate_osi, + validate_osi_ontology, +) + +__version__ = "0.1.0" + +__all__ = [ + "OBMLtoOSI", + "OBMLtoOSIOntology", + "OSItoOBML", + "ValidationResult", + "validate_obml", + "validate_osi", + "validate_osi_ontology", + "__version__", +] diff --git a/converters/orionbelt/src/ossie_orionbelt/_common.py b/converters/orionbelt/src/ossie_orionbelt/_common.py new file mode 100644 index 00000000..3460381f --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/_common.py @@ -0,0 +1,76 @@ +"""Shared constants and mapping tables for the OSI ↔ OBML converter. + +These module-level constants are used by more than one of the converter +direction classes (``OSItoOBML``, ``OBMLtoOSI``, ``OBMLtoOSIOntology``) and the +validation helpers. They live here so both the facade ``converter`` module and +the per-direction class modules can import them without forming an import cycle. +""" + +from __future__ import annotations + +import re + +# ─── Spec version pin ─────────────────────────────────────────────────────── +# Single source of truth for the OSI spec we emit. Bump when upstream cuts +# a stable v0.2.0 (drop the ``.dev0`` suffix). All read paths accept both +# 0.1.x (via the legacy shim) and 0.2.x. +_OSI_VERSION = "0.2.0.dev0" + +# SQL dialects (of the OSI enum) whose aggregation expressions our regex-based +# metric parser can read, in preference order. ANSI_SQL first; SNOWFLAKE and +# DATABRICKS are SQL engines OrionBelt also targets, and their simple/expression +# aggregations (``SUM(t.c)``, ``SUM(t.a * t.b)``) are syntactically identical to +# ANSI. MDX / TABLEAU / MAQL are non-SQL languages and are never parsed as SQL. +_SQL_PARSEABLE_DIALECTS = ("ANSI_SQL", "SNOWFLAKE", "DATABRICKS") + +# Matches a ``dataset.column`` reference inside a SQL expression, where each +# side is a bare identifier or a quoted identifier (double quotes, backticks, or +# brackets). The leading lookbehind prevents matching the tail of a longer path +# (``a.b.c``) or a mid-token boundary; the bare form must start with a letter or +# underscore so numeric literals (``1.5``) are never treated as references. +_COLUMN_REF_RE = re.compile( + r'(?[A-Za-z_]\w*|"[^"]+"|`[^`]+`|\[[^\]]+\])' + r"\s*\.\s*" + r'(?P[A-Za-z_]\w*|"[^"]+"|`[^`]+`|\[[^\]]+\])' +) +# Vendor identities for custom_extensions. +# ORIONBELT - OrionBelt/OBML-proprietary payloads we author on OBML -> OSI. +# OSI - OSI-native fields OBML can't hold (unique_keys, field label, +# ai_context leftovers), stashed into OBML on OSI -> OBML. +# Read paths also accept the legacy tags we emitted before this scheme so older +# documents still round-trip; foreign vendors (SNOWFLAKE, DBT, ...) are +# preserved verbatim, never relabelled. +_VENDOR_OBML = "ORIONBELT" +_VENDOR_OSI = "OSI" +_OBML_VENDOR_READ = ("ORIONBELT", "COMMON") +_OSI_VENDOR_READ = ("OSI", "OBSL") +# Vendors the converter handles internally (its own payloads + native-field +# stashes). Any custom_extension from a vendor outside this set is third-party +# and is carried through verbatim in both directions, never relabelled. +_INTERNAL_VENDORS = frozenset({"ORIONBELT", "COMMON", "OSI", "OBSL"}) + +# ─── Type mapping ─────────────────────────────────────────────────────────── + +OBML_TO_OSI_TYPE = { + "string": "string", + "json": "string", + "int": "integer", + "float": "number", + "date": "date", + "time": "time", + "time_tz": "time", + "timestamp": "timestamp", + "timestamp_tz": "timestamp", + "boolean": "boolean", +} + +OSI_TO_OBML_TYPE = { + "string": "string", + "integer": "int", + "number": "float", + "date": "date", + "time": "time", + "timestamp": "timestamp", + "boolean": "boolean", +} diff --git a/converters/orionbelt/src/ossie_orionbelt/cli.py b/converters/orionbelt/src/ossie_orionbelt/cli.py new file mode 100644 index 00000000..196722f6 --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/cli.py @@ -0,0 +1,155 @@ +"""Command-line entry point for the OBML <-> OSI converter. + +A single ``ossie-orionbelt`` command with two format-named subcommands, mirroring +the OSI converter convention (e.g. ``osi-dbt msi-to-osi``): + + ossie-orionbelt obml-to-osi -i model.obml.yaml -o model.osi.yaml + ossie-orionbelt obml-to-osi --ontology -i model.obml.yaml -o model.ontology.yaml + ossie-orionbelt osi-to-obml -i model.osi.yaml -o model.obml.yaml + +Both subcommands print conversion warnings and a validation summary to stderr, +and exit non-zero when the produced document fails schema validation (unless +``--no-validate``). +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path +from typing import Any + +import yaml + +from ossie_orionbelt.converter import ( + OBMLtoOSI, + OBMLtoOSIOntology, + OSItoOBML, + validate_obml, + validate_osi, + validate_osi_ontology, +) + + +def _load(input_path: str) -> dict[str, Any]: + data = yaml.safe_load(Path(input_path).read_text()) + if not isinstance(data, dict): + print(f"Error: {input_path} is not a YAML mapping", file=sys.stderr) + raise SystemExit(2) + return data + + +def _emit(result: dict[str, Any], output: str) -> None: + output_yaml = yaml.dump( + result, default_flow_style=False, allow_unicode=True, sort_keys=False, width=120 + ) + Path(output).write_text(output_yaml) + print(f"Written to {output}", file=sys.stderr) + + +def _print_warnings(warnings: list[str]) -> None: + if warnings: + print("\nConversion warnings:", file=sys.stderr) + for w in warnings: + print(f" - {w}", file=sys.stderr) + + +def _report_validation(label: str, result: dict[str, Any], validate_fn: Any) -> bool: + """Validate ``result`` and print a summary. Return True if there are errors.""" + print(f"\nValidating {label}...", file=sys.stderr) + vr = validate_fn(result) + for line in vr.summary_lines(): + print(line, file=sys.stderr) + if vr.valid: + print(f"{label} is valid", file=sys.stderr) + return False + print(f"{label} has validation errors", file=sys.stderr) + return True + + +def _cmd_obml_to_osi(args: argparse.Namespace) -> int: + """OBML -> OSI core-spec (or, with --ontology, OSI ontology).""" + data = _load(args.input) + + validate_fn: Any + if args.ontology: + converter: Any = OBMLtoOSIOntology( + data, args.model_name, args.description, args.ai_instructions + ) + result = converter.convert() + validate_fn, label = validate_osi_ontology, "OSI ontology output" + else: + converter = OBMLtoOSI(data, args.model_name, args.description, args.ai_instructions) + result = converter.convert() + validate_fn, label = validate_osi, "OSI output" + + _emit(result, args.output) + _print_warnings(converter.warnings) + + if args.no_validate: + return 0 + return 1 if _report_validation(label, result, validate_fn) else 0 + + +def _cmd_osi_to_obml(args: argparse.Namespace) -> int: + """OSI core-spec -> OBML.""" + data = _load(args.input) + + converter = OSItoOBML(data, args.database, args.schema) + result = converter.convert() + + _emit(result, args.output) + _print_warnings(converter.warnings) + + if args.no_validate: + return 0 + return 1 if _report_validation("OBML output", result, validate_obml) else 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="ossie-orionbelt", + description="Convert between OrionBelt OBML and OSI YAML.", + ) + subparsers = parser.add_subparsers(dest="command", required=True) + + o2s = subparsers.add_parser("obml-to-osi", help="Convert OBML YAML → OSI YAML") + o2s.add_argument("-i", "--input", required=True, metavar="FILE", help="Path to OBML YAML") + o2s.add_argument( + "-o", "--output", required=True, metavar="FILE", help="Path for output OSI YAML" + ) + o2s.add_argument( + "--ontology", action="store_true", help="Emit an OSI ontology document instead of core-spec" + ) + o2s.add_argument( + "--model-name", default="semantic_model", metavar="NAME", help="OSI semantic model name" + ) + o2s.add_argument("--description", default="", metavar="TEXT", help="OSI model description") + o2s.add_argument( + "--ai-instructions", default="", metavar="TEXT", help="OSI ai_context instructions" + ) + o2s.add_argument("--no-validate", action="store_true", help="Skip output validation") + + s2o = subparsers.add_parser("osi-to-obml", help="Convert OSI YAML → OBML YAML") + s2o.add_argument("-i", "--input", required=True, metavar="FILE", help="Path to OSI YAML") + s2o.add_argument( + "-o", "--output", required=True, metavar="FILE", help="Path for output OBML YAML" + ) + s2o.add_argument( + "--database", default="ANALYTICS", metavar="NAME", help="Default database for OBML output" + ) + s2o.add_argument( + "--schema", default="PUBLIC", metavar="NAME", help="Default schema for OBML output" + ) + s2o.add_argument("--no-validate", action="store_true", help="Skip output validation") + + args = parser.parse_args(argv) + if args.command == "obml-to-osi": + return _cmd_obml_to_osi(args) + if args.command == "osi-to-obml": + return _cmd_osi_to_obml(args) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/converters/orionbelt/src/ossie_orionbelt/converter.py b/converters/orionbelt/src/ossie_orionbelt/converter.py new file mode 100644 index 00000000..9d08ff8b --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/converter.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +""" +OSI ↔ OBML Bidirectional Converter +=================================== +Converts between Open Semantic Interchange (OSI v0.2.0.dev0) YAML models +and OrionBelt Markup Language (OBML v1.0) YAML models. + +OSI v0.1.1 inputs are still accepted on read — the legacy shim +``_normalize_legacy_v01`` promotes pre-v0.2 custom_extensions into the +v0.2 first-class fields before regular parsing runs. + +Author: OrionBelt / RALFORION + +This module is a thin **facade**. The converter implementation is split across +sibling modules to keep each file focused: + +* :mod:`ossie_orionbelt._common` — shared constants and mapping tables +* :mod:`ossie_orionbelt.osi_to_obml` — :class:`OSItoOBML` +* :mod:`ossie_orionbelt.obml_to_osi` — :class:`OBMLtoOSI` +* :mod:`ossie_orionbelt.ontology` — :class:`OBMLtoOSIOntology` +* :mod:`ossie_orionbelt.validation` — :class:`ValidationResult` + ``validate_*`` + +Every public name is re-exported here so ``ossie_orionbelt.converter.`` +continues to work unchanged. +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import yaml + +from ossie_orionbelt._common import ( + _COLUMN_REF_RE as _COLUMN_REF_RE, +) +from ossie_orionbelt._common import ( + _INTERNAL_VENDORS as _INTERNAL_VENDORS, +) +from ossie_orionbelt._common import ( + _OBML_VENDOR_READ as _OBML_VENDOR_READ, +) +from ossie_orionbelt._common import ( + _OSI_VENDOR_READ as _OSI_VENDOR_READ, +) +from ossie_orionbelt._common import ( + _OSI_VERSION as _OSI_VERSION, +) +from ossie_orionbelt._common import ( + _SQL_PARSEABLE_DIALECTS as _SQL_PARSEABLE_DIALECTS, +) +from ossie_orionbelt._common import ( + _VENDOR_OBML as _VENDOR_OBML, +) +from ossie_orionbelt._common import ( + _VENDOR_OSI as _VENDOR_OSI, +) + +# Shared constants / mapping tables (re-exported for backwards compatibility). +# The ``X as X`` aliases mark these as intentional re-exports so historic +# ``from ossie_orionbelt.converter import `` imports keep working. +from ossie_orionbelt._common import ( + OBML_TO_OSI_TYPE as OBML_TO_OSI_TYPE, +) +from ossie_orionbelt._common import ( + OSI_TO_OBML_TYPE as OSI_TO_OBML_TYPE, +) +from ossie_orionbelt.obml_to_osi import OBMLtoOSI as OBMLtoOSI +from ossie_orionbelt.ontology import OBMLtoOSIOntology as OBMLtoOSIOntology +from ossie_orionbelt.osi_to_obml import OSItoOBML as OSItoOBML +from ossie_orionbelt.validation import ( + _OBML_SCHEMA_PATH as _OBML_SCHEMA_PATH, +) +from ossie_orionbelt.validation import ( + _OSI_SCHEMA_PATH as _OSI_SCHEMA_PATH, +) +from ossie_orionbelt.validation import ( + _SCHEMAS_DIR as _SCHEMAS_DIR, +) +from ossie_orionbelt.validation import ( + _SCRIPT_DIR as _SCRIPT_DIR, +) +from ossie_orionbelt.validation import ( + ValidationResult as ValidationResult, +) +from ossie_orionbelt.validation import ( + _validate_json_schema as _validate_json_schema, +) +from ossie_orionbelt.validation import ( + validate_obml as validate_obml, +) +from ossie_orionbelt.validation import ( + validate_osi as validate_osi, +) +from ossie_orionbelt.validation import ( + validate_osi_ontology as validate_osi_ontology, +) + +__all__ = [ + "OBML_TO_OSI_TYPE", + "OSI_TO_OBML_TYPE", + "OBMLtoOSI", + "OBMLtoOSIOntology", + "OSItoOBML", + "ValidationResult", + "validate_obml", + "validate_osi", + "validate_osi_ontology", + "main", +] + + +# ═══════════════════════════════════════════════════════════════════════════ +# CLI Entry Point +# ═══════════════════════════════════════════════════════════════════════════ + + +def main(): + parser = argparse.ArgumentParser(description="OSI ↔ OBML Bidirectional Converter") + parser.add_argument("direction", choices=["osi2obml", "obml2osi"], help="Conversion direction") + parser.add_argument("input", nargs="?", help="Input YAML file") + parser.add_argument("-o", "--output", help="Output YAML file") + parser.add_argument( + "--name", default="semantic_model", help="Model name for OBML→OSI conversion" + ) + parser.add_argument( + "--description", default="", help="Model description for OBML→OSI conversion" + ) + parser.add_argument( + "--ai-instructions", default="", help="AI instructions for OBML→OSI conversion" + ) + parser.add_argument( + "--database", default="ANALYTICS", help="Default database for OSI→OBML conversion" + ) + parser.add_argument("--schema", default="PUBLIC", help="Default schema for OSI→OBML conversion") + parser.add_argument( + "--no-validate", action="store_true", help="Skip OBML validation after conversion" + ) + + args = parser.parse_args() + + if not args.input: + parser.error("Input file is required for conversion") + + input_path = Path(args.input) + with open(input_path) as f: + data = yaml.safe_load(f) + + if args.direction == "osi2obml": + converter = OSItoOBML(data, args.database, args.schema) + result = converter.convert() + warnings = converter.warnings + else: + converter = OBMLtoOSI(data, args.name, args.description, args.ai_instructions) + result = converter.convert() + warnings = converter.warnings + + # Output + output_yaml = yaml.dump( + result, default_flow_style=False, allow_unicode=True, sort_keys=False, width=120 + ) + + if args.output: + with open(args.output, "w") as f: + f.write(output_yaml) + print(f"✅ Converted to {args.output}") + else: + print(output_yaml) + + if warnings: + print("\n⚠️ Conversion warnings:", file=sys.stderr) + for w in warnings: + print(f" - {w}", file=sys.stderr) + + # ── Validate output ──────────────────────────────────────────────── + if not args.no_validate: + has_errors = False + + if args.direction == "osi2obml": + # Validate OBML output + print("\n🔍 Validating OBML output...", file=sys.stderr) + vr = validate_obml(result) + for line in vr.summary_lines(): + print(line, file=sys.stderr) + if vr.valid: + print("✅ OBML output is valid", file=sys.stderr) + else: + print("❌ OBML output has validation errors", file=sys.stderr) + has_errors = True + else: + # Validate OBML input (source) and OSI output + print("\n🔍 Validating OBML input...", file=sys.stderr) + vr_obml = validate_obml(data) + for line in vr_obml.summary_lines(): + print(line, file=sys.stderr) + if vr_obml.valid: + print("✅ OBML input is valid", file=sys.stderr) + else: + print("❌ OBML input has validation errors", file=sys.stderr) + has_errors = True + + print("\n🔍 Validating OSI output...", file=sys.stderr) + vr_osi = validate_osi(result) + for line in vr_osi.summary_lines(): + print(line, file=sys.stderr) + if vr_osi.valid: + print("✅ OSI output is valid", file=sys.stderr) + else: + print("❌ OSI output has validation errors", file=sys.stderr) + has_errors = True + + if has_errors: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/converters/orionbelt/src/ossie_orionbelt/obml_to_osi.py b/converters/orionbelt/src/ossie_orionbelt/obml_to_osi.py new file mode 100644 index 00000000..9f3dacaf --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/obml_to_osi.py @@ -0,0 +1,1124 @@ +"""OBML → OSI conversion (the :class:`OBMLtoOSI` direction). + +Extracted verbatim from ``converter.py``; see that module for the package-level +docstring and the shared constants in :mod:`ossie_orionbelt._common`. +""" + +from __future__ import annotations + +import json +import re +from typing import Any + +from ossie_orionbelt._common import ( + _INTERNAL_VENDORS, + _OSI_VENDOR_READ, + _OSI_VERSION, + _VENDOR_OBML, + OBML_TO_OSI_TYPE, +) + + +class OBMLtoOSI: + """Convert an OBML semantic model YAML to OSI format.""" + + def __init__( + self, + obml: dict, + model_name: str = "semantic_model", + model_description: str = "", + ai_instructions: str = "", + ): + self.obml = obml + self.model_name = model_name + self.model_description = model_description + self.ai_instructions = ai_instructions + self.warnings: list[str] = [] + + def convert(self) -> dict: + # Reset warnings so a second convert() call on the same instance does + # not duplicate them. + self.warnings = [] + + osi: dict[str, Any] = {"version": _OSI_VERSION} + + data_objects = self.obml.get("dataObjects", {}) + obml_dimensions = self.obml.get("dimensions", {}) + obml_measures = self.obml.get("measures", {}) + obml_metrics = self.obml.get("metrics", {}) + + # ── Datasets ──────────────────────────────────────────────── + datasets = [] + all_relationships = [] + + for do_name, do_obj in data_objects.items(): + dataset, rels = self._convert_data_object(do_name, do_obj, obml_dimensions) + datasets.append(dataset) + all_relationships.extend(rels) + + # ── Metrics (OBML measures + metrics → OSI metrics) ──────── + osi_metrics = self._convert_measures_and_metrics(obml_measures, obml_metrics, data_objects) + + # Re-emit OSI metrics that OBML could not represent and that the import + # path preserved verbatim (vendor OSI, ``obml_unconverted_metrics``). + # This closes the OSI -> OBML -> OSI roundtrip for non-SQL or + # non-decomposable metrics. + self._merge_restored_metrics(osi_metrics) + + # ── Build semantic model ──────────────────────────────────── + sem_model: dict[str, Any] = {"name": self.model_name} + # Prefer OBML model-level description, fall back to constructor param + obml_description = self.obml.get("description", "") + model_desc = obml_description or self.model_description + if model_desc: + sem_model["description"] = model_desc + if self.ai_instructions: + sem_model["ai_context"] = {"instructions": self.ai_instructions} + + sem_model["datasets"] = datasets + + if all_relationships: + sem_model["relationships"] = all_relationships + + if osi_metrics: + sem_model["metrics"] = osi_metrics + + # Add OBML as custom extension for lossless roundtrip info + roundtrip_data: dict[str, Any] = { + "source_format": "OBML", + "source_version": str(self.obml.get("version", "1.0")), + "converter": "ossie-orionbelt", + } + # Preserve model-level static filters for roundtrip + obml_filters = self.obml.get("filters", []) + if obml_filters: + roundtrip_data["obml_filters"] = obml_filters + # Preserve model settings for roundtrip + obml_settings = self.obml.get("settings") + if obml_settings: + roundtrip_data["obml_settings"] = obml_settings + # Preserve model owner for roundtrip + obml_owner = self.obml.get("owner") + if obml_owner: + roundtrip_data["obml_owner"] = obml_owner + # Preserve count-synthesis knobs (OSI has no native equivalent). The + # synthesized ``.count`` measures themselves are NOT emitted — + # they are derived and regenerate on load — but the knobs must survive + # a roundtrip. ``is not None`` so an explicit ``exposeCounts: false`` is + # preserved (``False`` is falsy). + expose_counts = self.obml.get("exposeCounts") + if expose_counts is not None: + roundtrip_data["obml_expose_counts"] = expose_counts + count_label_pattern = self.obml.get("countLabelPattern") + if count_label_pattern is not None: + roundtrip_data["obml_count_label_pattern"] = count_label_pattern + sem_model["custom_extensions"] = [ + { + "vendor_name": _VENDOR_OBML, + "data": json.dumps(roundtrip_data), + } + ] + # Re-emit third-party model-level vendor extensions verbatim + self._emit_foreign_extensions( + self.obml.get("customExtensions"), sem_model["custom_extensions"] + ) + + osi["semantic_model"] = [sem_model] + + # The published OSI core schema forbids root-level ``dialects`` / + # ``vendors`` (root is additionalProperties:false, only ``version`` + + # ``semantic_model``). Dialects live per-expression in + # ``expression.dialects[]`` and vendors per-entity in + # ``custom_extensions[].vendor_name`` — the schema-valid homes — so the + # document stays fully conformant without root advertisement arrays. + # See OSI PR #148 (and the single-document-dialect direction in #52). + return osi + + def _emit_foreign_extensions(self, obml_exts: list[dict] | None, osi_exts: list[dict]) -> None: + """Re-emit third-party OBML customExtensions as OSI custom_extensions. + + Mirrors ``OSItoOBML._carry_foreign_extensions``: extensions from a + vendor we do not handle internally are passed back to OSI under their + original vendor name, completing the roundtrip. + """ + for ext in obml_exts or []: + vendor = ext.get("vendor") + if vendor and vendor not in _INTERNAL_VENDORS: + osi_exts.append({"vendor_name": vendor, "data": ext.get("data", "")}) + + def _convert_data_object( + self, do_name: str, do_obj: dict, obml_dimensions: dict + ) -> tuple[dict, list]: + """Convert an OBML dataObject to an OSI dataset + relationships.""" + database = do_obj.get("database", "") + schema = do_obj.get("schema", "") + code = do_obj.get("code", "") + source = f"{database}.{schema}.{code}" if database else code + + # Use the OBML display name as the OSI dataset name so that + # relationship references (joinTo) stay consistent in roundtrips + osi_name = do_name + + dataset: dict[str, Any] = { + "name": osi_name, + "source": source, + } + + # ── Primary key (v0.2 first-class) ────────────────────────── + # Collect columns flagged with ``primaryKey: true`` in OBML order + # (TrackedLoader / Python dict preserves declaration order, which + # is significant for composite PKs). + pk_columns = [ + col_name + for col_name, col in (do_obj.get("columns", {}) or {}).items() + if col.get("primaryKey") + ] + # Use the physical ``code`` for each column when present — OSI + # field names mirror the physical column code (see _convert_column). + if pk_columns: + columns_map = do_obj.get("columns", {}) or {} + dataset["primary_key"] = [ + columns_map[c].get("code", c.lower().replace(" ", "_")) for c in pk_columns + ] + + # ── Unique keys (v0.2 first-class, lossless roundtrip via OBSL) ── + # OBML doesn't model unique keys natively today; round-trip via the + # ``OBSL``-vendor ``obml_unique_keys`` payload that originated from + # a prior OSI → OBML conversion (or hand-authored OBML). + unique_keys_extra: list[list[str]] | None = None + for ext in do_obj.get("customExtensions", []) or []: + if ext.get("vendor") not in _OSI_VENDOR_READ: + continue + try: + data = json.loads(ext.get("data", "{}")) + except (json.JSONDecodeError, TypeError): + continue + uk = data.get("obml_unique_keys") + if isinstance(uk, list) and all(isinstance(g, list) for g in uk): + unique_keys_extra = [list(g) for g in uk] + break + if unique_keys_extra: + dataset["unique_keys"] = unique_keys_extra + + if do_obj.get("description"): + dataset["description"] = do_obj["description"] + elif do_obj.get("comment"): + dataset["description"] = do_obj["comment"] + + # ── Rebuild ai_context: native synonyms + remaining from customExtensions + ai_ctx: dict[str, Any] = {} + for ext in do_obj.get("customExtensions", []): + if ext.get("vendor") == "OSI": + try: + ai_data = json.loads(ext.get("data", "{}")) + if ai_data: + ai_ctx.update(ai_data) + except (json.JSONDecodeError, TypeError): + pass + # Merge native OBML synonyms into ai_context.synonyms + obml_synonyms = do_obj.get("synonyms", []) + if obml_synonyms: + existing = ai_ctx.get("synonyms", []) + merged = list(existing) + [s for s in obml_synonyms if s not in existing] + ai_ctx["synonyms"] = merged + if ai_ctx: + dataset["ai_context"] = ai_ctx + + # ── Fields ────────────────────────────────────────────────── + fields = [] + columns = do_obj.get("columns", {}) + for col_name, col_obj in columns.items(): + field = self._convert_column(col_name, col_obj, do_name, obml_dimensions) + fields.append(field) + + if fields: + dataset["fields"] = fields + + # ── Relationships (from OBML joins) ───────────────────────── + relationships = [] + joins = do_obj.get("joins", []) + for i, join in enumerate(joins): + rel = self._convert_join_to_relationship(osi_name, do_name, do_obj, join, i) + if rel: + relationships.append(rel) + + # ── Preserve DataObject owner/comment + refresh in custom_extensions ── + do_extras: dict[str, Any] = {} + if do_obj.get("owner"): + do_extras["obml_owner"] = do_obj["owner"] + if do_obj.get("comment"): + do_extras["obml_comment"] = do_obj["comment"] + # OBML-only freshness contract — round-tripped through OSI + # custom_extensions since OSI has no native equivalent. See + # design/PLAN_freshness_driven_cache.md §5. + if do_obj.get("refresh"): + do_extras["obml_refresh"] = do_obj["refresh"] + # Count-synthesis knobs (``is not None`` so ``countable: false`` survives). + if do_obj.get("countable") is not None: + do_extras["obml_countable"] = do_obj["countable"] + if do_obj.get("countLabel") is not None: + do_extras["obml_count_label"] = do_obj["countLabel"] + if do_extras: + ds_exts = dataset.setdefault("custom_extensions", []) + ds_exts.append( + { + "vendor_name": _VENDOR_OBML, + "data": json.dumps(do_extras), + } + ) + + # Re-emit third-party vendor extensions verbatim + self._emit_foreign_extensions( + do_obj.get("customExtensions"), dataset.setdefault("custom_extensions", []) + ) + if not dataset["custom_extensions"]: + del dataset["custom_extensions"] + + return dataset, relationships + + def _convert_column( + self, col_name: str, col_obj: dict, do_name: str, obml_dimensions: dict + ) -> dict: + """Convert an OBML column to an OSI field.""" + code = col_obj.get("code", col_name.lower().replace(" ", "_")) + + field: dict[str, Any] = { + "name": code, + "expression": { + "dialects": [ + { + "dialect": "ANSI_SQL", + "expression": code, + } + ] + }, + } + + # Check if this column is used as a dimension + is_dimension = False + is_time = False + synonyms = [] + + for dim_name, dim_obj in obml_dimensions.items(): + if dim_obj.get("dataObject") == do_name and dim_obj.get("column") == col_name: + is_dimension = True + if dim_obj.get("resultType") in ("date", "time", "timestamp", "timestamp_tz"): + is_time = True + # The dimension display name is a synonym + if dim_name != col_name: + synonyms.append(dim_name) + break + + abstract_type = col_obj.get("abstractType", "string") + if abstract_type in ("date", "timestamp", "timestamp_tz"): + is_time = True + + if is_dimension or is_time: + field["dimension"] = {"is_time": is_time} + + if col_obj.get("description"): + field["description"] = col_obj["description"] + elif col_obj.get("comment"): + field["description"] = col_obj["comment"] + else: + field["description"] = col_name # Use display name as description + + # ── Field label (OSI v0.2 first-class) ── + # Surfaced from OBSL-vendor customExtensions ``obml_field_label`` — + # round-trip path for OSI → OBML → OSI fidelity. OBML has no + # native column ``label`` today. + for ext in col_obj.get("customExtensions", []) or []: + if ext.get("vendor") not in _OSI_VENDOR_READ: + continue + try: + ext_label_data = json.loads(ext.get("data", "{}")) + except (json.JSONDecodeError, TypeError): + continue + if ext_label_data.get("obml_field_label"): + field["label"] = ext_label_data["obml_field_label"] + break + + # Restore ai_context from customExtensions (OSI vendor) if present + ai_ctx: dict[str, Any] = {} + for ext in col_obj.get("customExtensions", []): + if ext.get("vendor") == "OSI": + try: + ai_data = json.loads(ext.get("data", "{}")) + if ai_data: + ai_ctx.update(ai_data) + except (json.JSONDecodeError, TypeError): + pass + + # Merge native OBML column synonyms into ai_context + obml_col_synonyms = col_obj.get("synonyms", []) + if obml_col_synonyms: + existing = ai_ctx.get("synonyms", []) + merged = list(existing) + [s for s in obml_col_synonyms if s not in existing] + ai_ctx["synonyms"] = merged + + # Build ai_context with synonyms from display name + display_synonym = col_name.lower() + code_clean = code.lower() + if display_synonym != code_clean: + synonyms.insert(0, col_name) + if synonyms: + ai_ctx.setdefault("synonyms", []).extend( + s for s in synonyms if s not in ai_ctx.get("synonyms", []) + ) + if ai_ctx: + field["ai_context"] = ai_ctx + + # Preserve OBML type info in custom_extensions for roundtrip fidelity + abstract_type = col_obj.get("abstractType", "string") + osi_type = OBML_TO_OSI_TYPE.get(abstract_type, "string") + ext_data: dict[str, Any] = { + "data_type": osi_type, + "obml_abstract_type": abstract_type, + } + # Preserve OBML-only column properties + if col_obj.get("sqlType"): + ext_data["obml_sql_type"] = col_obj["sqlType"] + if col_obj.get("sqlPrecision") is not None: + ext_data["obml_sql_precision"] = col_obj["sqlPrecision"] + if col_obj.get("sqlScale") is not None: + ext_data["obml_sql_scale"] = col_obj["sqlScale"] + if col_obj.get("numClass"): + ext_data["obml_num_class"] = col_obj["numClass"] + if col_obj.get("comment"): + ext_data["obml_comment"] = col_obj["comment"] + if col_obj.get("owner"): + ext_data["obml_owner"] = col_obj["owner"] + # Preserve OBML-only dimension properties (timeGrain, format, resultType, etc.) + matched_dim: dict[str, Any] | None = None + for _dim_name, dim_obj in obml_dimensions.items(): + if dim_obj.get("dataObject") == do_name and dim_obj.get("column") == col_name: + matched_dim = dim_obj + if dim_obj.get("timeGrain"): + ext_data["obml_time_grain"] = dim_obj["timeGrain"] + if dim_obj.get("format"): + ext_data["obml_dimension_format"] = dim_obj["format"] + if dim_obj.get("resultType"): + ext_data["obml_dimension_result_type"] = dim_obj["resultType"] + if dim_obj.get("description"): + ext_data["obml_dimension_description"] = dim_obj["description"] + if dim_obj.get("owner"): + ext_data["obml_dimension_owner"] = dim_obj["owner"] + if dim_obj.get("via"): + ext_data["obml_dimension_via"] = dim_obj["via"] + break + field["custom_extensions"] = [ + { + "vendor_name": _VENDOR_OBML, + "data": json.dumps(ext_data), + } + ] + + # Re-emit third-party vendor extensions verbatim. OSI has no separate + # dimension entity, so a matched dimension's foreign extensions surface + # on the field too (they re-import onto the column). + self._emit_foreign_extensions(col_obj.get("customExtensions"), field["custom_extensions"]) + if matched_dim is not None: + self._emit_foreign_extensions( + matched_dim.get("customExtensions"), field["custom_extensions"] + ) + + return field + + def _convert_join_to_relationship( + self, osi_from_name: str, _obml_from_name: str, from_do: dict, join: dict, index: int + ) -> dict | None: + """Convert an OBML join to an OSI relationship.""" + join_to_display = join.get("joinTo", "") + # Use the OBML display name as the OSI target name (consistent with + # _convert_data_object which uses display name as OSI dataset name) + to_name = join_to_display + target_do = self.obml.get("dataObjects", {}).get(join_to_display, {}) + + # Map column display names to codes + from_columns_display = join.get("columnsFrom", []) + to_columns_display = join.get("columnsTo", []) + + from_cols = self._resolve_column_codes(from_do, from_columns_display) + to_cols = self._resolve_column_codes(target_do, to_columns_display) + + # Generate relationship name + path_name = join.get("pathName", "") + if path_name: + rel_name = f"{osi_from_name}_to_{to_name}_{path_name}" + else: + rel_name = f"{osi_from_name}_to_{to_name}" + if index > 0: + rel_name += f"_{index}" + + rel: dict[str, Any] = { + "name": rel_name, + "from": osi_from_name, + "to": to_name, + "from_columns": from_cols, + "to_columns": to_cols, + } + + # Preserve secondary join info in ai_context + if join.get("secondary"): + rel["ai_context"] = { + "instructions": ( + f"Secondary/alternative join path" + f"{(' named: ' + path_name) if path_name else ''}. " + f"Use only when explicitly needed." + ) + } + + return rel + + def _resolve_column_codes(self, do_obj: dict, col_display_names: list) -> list: + """Resolve OBML column display names to their code values.""" + columns = do_obj.get("columns", {}) + codes = [] + for display in col_display_names: + col = columns.get(display, {}) + codes.append(col.get("code", display.lower().replace(" ", "_"))) + return codes + + def _restore_unconverted_metrics(self) -> list[dict]: + """Recover OSI metrics preserved verbatim during OSI -> OBML import. + + The import path stashes metrics OBML can't represent under an OSI-vendor + model-level customExtension (``obml_unconverted_metrics``). Re-emit them + unchanged so a full OSI -> OBML -> OSI roundtrip keeps them. + """ + restored: list[dict] = [] + for ext in self.obml.get("customExtensions", []) or []: + if ext.get("vendor") not in _OSI_VENDOR_READ: + continue + try: + data = json.loads(ext.get("data", "{}")) + except (json.JSONDecodeError, TypeError): + continue + preserved = data.get("obml_unconverted_metrics") + if isinstance(preserved, list): + restored.extend(m for m in preserved if isinstance(m, dict)) + return restored + + def _merge_restored_metrics(self, osi_metrics: list[dict]) -> None: + """Append preserved (unconverted) OSI metrics to the converted ones. + + Name-collision guard: if a queryable OBML measure/metric now owns a name + a stale preserved metric also uses, skip the preserved copy (the real + metric wins) so the OSI output has no duplicate metric names and passes + semantic validation. Each appended metric carries its own dialects + (``expression.dialects[]``) and vendors (``custom_extensions``) + verbatim, which are the schema-valid homes for that metadata. + """ + existing = {m.get("name") for m in osi_metrics if isinstance(m, dict)} + for restored in self._restore_unconverted_metrics(): + name = restored.get("name") + if name in existing: + self.warnings.append( + f"Preserved OSI metric '{name}' dropped on export: a converted " + f"OBML metric now uses that name." + ) + continue + existing.add(name) + osi_metrics.append(restored) + + def _convert_measures_and_metrics( + self, obml_measures: dict, obml_metrics: dict, data_objects: dict + ) -> list: + """Convert OBML measures and metrics to OSI metrics.""" + osi_metrics = [] + + # Convert each OBML measure to an OSI metric + for measure_name, measure_obj in obml_measures.items(): + osi_metric = self._convert_measure(measure_name, measure_obj, data_objects) + if osi_metric: + self._carry_foreign_to_osi_metric(measure_obj, osi_metric) + osi_metrics.append(osi_metric) + + # Convert OBML metrics (which reference measures) to OSI metrics + for metric_name, metric_obj in obml_metrics.items(): + if metric_obj.get("type") == "cumulative": + osi_metric = self._convert_obml_cumulative_metric( + metric_name, metric_obj, obml_measures, data_objects + ) + elif metric_obj.get("type") == "period_over_period": + osi_metric = self._convert_obml_pop_metric( + metric_name, metric_obj, obml_measures, data_objects + ) + elif metric_obj.get("type") == "window": + osi_metric = self._convert_obml_window_metric( + metric_name, metric_obj, obml_measures, data_objects + ) + else: + osi_metric = self._convert_obml_metric( + metric_name, metric_obj, obml_measures, data_objects + ) + if osi_metric: + self._carry_foreign_to_osi_metric(metric_obj, osi_metric) + osi_metrics.append(osi_metric) + + return osi_metrics + + def _carry_foreign_to_osi_metric(self, obml_obj: dict, osi_metric: dict) -> None: + """Re-emit third-party vendor extensions on an OBML measure/metric to + the OSI metric, dropping the key again if nothing foreign was added.""" + self._emit_foreign_extensions( + obml_obj.get("customExtensions"), osi_metric.setdefault("custom_extensions", []) + ) + if not osi_metric["custom_extensions"]: + del osi_metric["custom_extensions"] + + def _convert_measure(self, name: str, measure: dict, data_objects: dict) -> dict | None: + """Convert an OBML measure to an OSI metric.""" + + columns = measure.get("columns", []) + agg = measure.get("aggregation", "sum").upper() + distinct = measure.get("distinct", False) + obml_synonyms = measure.get("synonyms", []) + + # Build ai_context with synonyms (name + native OBML synonyms) + ai_synonyms = [name] + [s for s in obml_synonyms if s != name] + ai_ctx: dict[str, Any] = {"synonyms": ai_synonyms} if ai_synonyms else {} + + # ``aggregation: measure`` delegates resolution to the engine + # (Databricks Metric View) — there is no source column to read + # and no ANSI_SQL expression to emit. OSI has no first-class + # concept for engine-delegated aggregation, so we serialize the + # measure as an OSI metric whose expression is the literal + # ``MEASURE("