Add datatype field to Field and Metric; reframe is_time as role marker

[jonmmease]

AI Disclosure: I drove the original PR and this updated port iteratively with Claude Code and Codex, and have reviewed it fully.

Motivation

This ports and extends open-semantic-interchange/ossie#113 for the current Apache Ossie codebase.

The original proposal grew out of open-semantic-interchange/ossie#25, which became open-semantic-interchange/ossie#84, and is also related to open-semantic-interchange/ossie#17.

The issue originally framed this as “use datatype rather than is_time.” After looking at existing semantic-layer implementations, the cleaner model was to represent data type and temporal role independently:

• datatype says what values a field or metric produces.
• dimension.is_time says whether a field plays a temporal-dimension role.

For example, d_year, d_quarter_name, and d_month_name may have different scalar types while all serving temporal roles. Snowflake Semantic Views and LookML similarly separate scalar representation from temporal classification.

This PR therefore adds datatype alongside is_time, then carries that new metadata through the Python bindings and the existing converter paths that have native type information.

What changes

Core specification

Adds an optional top-level datatype property to both Field and Metric, backed by a logical-type vocabulary:

• String
• Integer
• Decimal
• Float
• Boolean
• Date
• Time
• DateTime
• DateTimeTz
• Opaque

These types have been updated since the original PR to match those introduced by the ontology specification. So the prior number is split into Integer/Float/Decimal. They also include three types that are not part of ontology type system, but I'd be happy to add them there as well for consistency: Time, DateTimeTz, and Opaque.

These are logical exchange types, not SQL-physical types. In particular:

• Decimal is exact base-10 with unspecified precision and scale; Float is approximate.
• DateTime has no timezone or offset; DateTimeTz identifies an instant using offset or timezone context, without promising preservation of a named timezone.
• Unknown or unspecified types should omit datatype.
• A known type outside the vocabulary should use Opaque plus custom_extensions when an exact vendor type needs to be retained.

Temporal role semantics

dimension.is_time remains valid and is documented as independent of datatype.

For fields with a dimension block:

1. An explicit is_time value wins.
2. When is_time is unset, it defaults to true for Date, Time, DateTime, and DateTimeTz, and to false otherwise.

A field without dimension metadata remains a fact regardless of its datatype. This supports combinations such as:

• datatype: Integer plus is_time: true for a year grain.
• datatype: DateTimeTz plus is_time: false for an audit timestamp that should not be exposed as a time dimension.
• is_time: true without datatype for legacy models that know the role but do not declare a scalar type.

Python bindings

• Adds OSIDataType and exports it from the package.
• Adds optional datatype fields to OSIField and OSIMetric.
• Adds OSIField.is_time_dimension() for the effective role rule above.
• Adds POLARIS to the supported custom-extension vendor constants so Polaris-imported models validate in Python.
• Tests keep the Python enum synchronized with the core JSON Schema.

Existing converters

This PR makes targetted updates to existing import/export logic;

Converter	Direction	Datatype behavior
Snowflake	Ossie → Snowflake	Emits field data_type for portable mappings, including DateTime → TIMESTAMP_NTZ and DateTimeTz → TIMESTAMP_TZ; omits Opaque with a warning. Field classification also follows the explicit-role/temporal-default rule. Snowflake metric result types remain inferred from expressions.
GoodData	GoodData ↔ Ossie	Maps explicitly present sourceColumnDataType values to and from field datatype; omitted source types remain unspecified. Unknown GoodData types become Opaque and retain their exact value in a GOODDATA extension. GoodData metrics remain outside the existing converter's scope.
Polaris	Polaris ↔ Ossie	Maps Iceberg column types to logical datatypes and maps Ossie datatypes to default Iceberg types. Exact Iceberg type JSON is retained in a POLARIS extension so integer width, decimal precision/scale, timestamp precision, and nested structures can round-trip. Nested Iceberg field IDs are regenerated on export. Polaris catalog conversion does not currently convert metrics.
Salesforce	Salesforce ↔ Ossie fields; Salesforce → Ossie metrics	Maps field types in both existing directions and imports calculated-measurement result types onto Ossie metrics. Exact Salesforce types are retained in SALESFORCE extensions so distinctions such as Email/Text and Currency/Number survive round trips. Exporting timezone-free DateTime warns because Salesforce's single DateTime type re-imports as DateTimeTz. Existing Ossie → Salesforce metric export remains unsupported.
dbt MetricFlow	Ossie → MetricFlow role classification	Uses the effective is_time rule when classifying MetricFlow time dimensions, including the temporal-datatype default and explicit opt-outs. MetricFlow → Ossie still cannot populate datatypes because the semantic manifest lacks column type metadata.

When an exact vendor type in custom_extensions conflicts with a portable datatype, the GoodData, Polaris, and Salesforce exporters preserve the exact vendor value and warn. This prioritizes lossless round trips over a lossy default mapping.

Examples and documentation

• Updates examples/tpcds_semantic_model.yaml: 29 of its 31 fields and all 5 metrics now declare datatypes. It retains examples of datatype without an explicit time role, datatype plus is_time, and legacy is_time without datatype.
• Documents the type-versus-role split, default rule, consumer guidance, migration behavior, and implementation precedent in the core specification.
• Updates converter guidance and converter-specific mapping references.
• Updates the public glossary's Field entry.
• Ignores Python __pycache__ directories.

Compatibility and impact

The core schema change is additive: datatype is optional on fields and metrics.

• Existing models that use only is_time continue to validate and preserve their temporal roles.
• Untyped fields continue to use each converter's existing fallback behavior.
• New models can declare logical types without conflating them with dimension/fact or temporal-role classification.
• An explicit is_time: false opts a temporal-typed dimension out of time-dimension treatment.
• Physical type details that do not fit the portable vocabulary are retained in vendor extensions where the source converter exposes them.

The dbt converter does not infer datatypes during MetricFlow → Ossie conversion. MetricFlow's semantic manifest does not provide the required column type metadata; supporting that direction would require a separate catalog or manifest input rather than guessing from semantic roles or expressions. Ossie → MetricFlow does honor the effective is_time rule because the Ossie input already carries the necessary datatype and dimension metadata.

To be conservative, this PR also does not add Time or DateTimeTz to the ontology's built-in value types, but I'm happy to add them for consistency if desired.

---

Since this PR updates datatype handling across several existing converters, I wanted to specifically ping the people who've worked on those implementations: @sfc-gh-tgao for Snowflake, @jaceksan for GoodData, @jbonofre for Polaris, and @deutschn for Salesforce. Would welcome any level of review or feedback.

[Copilot]

Pull request overview

This PR extends the Ossie core spec and implementations to support an optional, portable logical datatype on both Field and Metric, while reframing dimension.is_time as an independent temporal-role marker whose effective value can default from temporal datatypes when unset. The change is carried through the JSON Schema, Python bindings, documentation/examples, and multiple converters (Snowflake, GoodData, Polaris, Salesforce, dbt MetricFlow).

Changes:

• Add DataType vocabulary and optional datatype fields to the core JSON Schema plus spec/docs, and update examples accordingly.
• Update Python models with OSIDataType, add OSIField.is_time_dimension() to implement the effective-role rule, and add tests to keep Python and schema enums synchronized.
• Update converter implementations/tests/docs to map vendor-native types ↔ portable datatype, while keeping datatype independent from dimension/fact classification and from dimension.is_time.

Reviewed changes

Copilot reviewed 42 out of 43 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
python/tests/test_models.py	Adds Python tests for datatype enum sync, serialization, validation, and effective time-role behavior.
python/src/ossie/models.py	Introduces OSIDataType, adds datatype to field/metric, and implements OSIField.is_time_dimension().
python/src/ossie/init.py	Exports OSIDataType from the Python package.
examples/tpcds_semantic_model.yaml	Annotates many fields/metrics with datatype and demonstrates type vs. time-role combinations.
docs/index.md	Updates glossary entry for Field to mention optional datatype and its vocabulary.
core-spec/spec.yaml	Documents datatype and clarifies dimension.is_time semantics and defaulting behavior.
core-spec/spec.md	Adds DataType section, documents type-vs-role split, updates field/metric reference tables and examples.
core-spec/osi-schema.json	Adds $defs.DataType enum and datatype properties on Field and Metric; updates is_time description.
converters/snowflake/tests/test_osi_to_snowflake_yaml_converter.py	Adds tests for datatype→Snowflake mapping and time-dimension classification defaulting from datatype.
converters/snowflake/tests/example_converted_tpcds_semantic_model.yaml	Updates expected Snowflake YAML output to include emitted data_type for fields where applicable.
converters/snowflake/src/osi_to_snowflake_yaml_converter.py	Implements _convert_datatype and uses datatype for field data_type emission and time-dimension defaults.
converters/snowflake/README.md	Documents Snowflake datatype mapping and clarifies that metric result types remain inferred.
converters/salesforce/src/test/resources/examples/osiToSalesforce.yaml	Updates Salesforce converter example input to include datatype on fields and metrics.
converters/salesforce/src/test/java/org/apache/ossie/SalesforceToOsiConverterTest.java	Extends Salesforce→Ossie tests to assert imported field/metric datatype and time dimension typing.
converters/salesforce/src/test/java/org/apache/ossie/OsiToSalesforceConverterTest.java	Adds assertions ensuring exact Salesforce extension types override portable datatypes on export.
converters/salesforce/src/test/java/org/apache/ossie/converter/SalesforceDataTypeMapperTest.java	Adds unit tests for Salesforce↔Ossie type mapping, compatibility, and timezone-lossy cases.
converters/salesforce/src/main/java/org/apache/ossie/converter/SalesforceDataTypeMapper.java	Introduces centralized Salesforce↔Ossie datatype mapping and compatibility/timezone-lossy helpers.
converters/salesforce/src/main/java/org/apache/ossie/converter/MetricMappingHandler.java	Imports Salesforce metric dataType into Ossie metric datatype during Salesforce→Ossie conversion.
converters/salesforce/src/main/java/org/apache/ossie/converter/FieldMappingHandler.java	Uses mapper for temporal detection and adds portable datatype export logic with precedence/warnings.
converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java	Adds datatype constant and removes now-replaced hardcoded Salesforce date/datetime constants.
converters/salesforce/README.md	Updates capability claims and adds detailed datatype + role mapping reference and limitations.
converters/polaris/src/test/java/org/apache/ossie/converter/polaris/OsiPolarisConverterTest.java	Extends tests to cover datatype import/export precedence, extension round-trips, and legacy fallbacks.
converters/polaris/src/test/java/org/apache/ossie/converter/polaris/IcebergTypeMapperTest.java	Adds unit tests for Iceberg physical type ↔ Ossie datatype mapping and nested ID regeneration.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/PolarisImporter.java	Maps Iceberg types to portable datatypes and preserves exact Iceberg type JSON in POLARIS extensions.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/PolarisExporter.java	Exports using exact extension type first, then portable datatype defaults, then legacy heuristics.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OsiYamlGenerator.java	Emits datatype and field-level custom_extensions in generated Ossie YAML.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OsiModelParser.java	Parses field datatype and reuses custom-extension parsing for datasets/fields.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/model/OsiModel.java	Adds datatype and customExtensions to the Polaris converter’s internal Field model.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/IcebergTypeMapper.java	Adds shared Iceberg physical type utilities and datatype mapping + exact-type export preparation.
converters/polaris/README.md	Documents datatype and exact Iceberg type extension behavior and precedence order.
converters/index.md	Updates converter guidance to describe datatype vs dimension.is_time semantics and defaulting.
converters/gooddata/tests/test_roundtrip.py	Adds round-trip tests for GoodData sourceColumnDataType ↔ Ossie datatype including Opaque extension cases.
converters/gooddata/tests/test_osi_to_gooddata.py	Adds tests for datatype mapping, precedence of exact extensions, warnings, and legacy date-dataset heuristic.
converters/gooddata/tests/test_models.py	Updates model tests to assert omitted source types remain None and aren’t serialized spuriously.
converters/gooddata/tests/test_gooddata_to_osi.py	Adds tests for GoodData→Ossie datatype mapping, Opaque extension retention, and omission behavior.
converters/gooddata/src/ossie_gooddata/osi_to_gooddata.py	Uses shared mapping helper; preserves legacy explicit-time heuristic for date datasets.
converters/gooddata/src/ossie_gooddata/models.py	Makes source_column_data_type optional and adjusts (de)serialization to omit defaults when unset.
converters/gooddata/src/ossie_gooddata/gooddata_to_osi.py	Maps native GoodData types into portable Ossie datatype and stores unknowns via Opaque extension.
converters/gooddata/src/ossie_gooddata/datatype_mapping.py	Adds shared GoodData↔Ossie datatype mapping with precedence and warning semantics.
converters/gooddata/README.md	Documents datatype mapping, precedence rules, and limitations (Float/Time/Opaque behaviors).
converters/dbt/tests/test_osi_to_msi.py	Adds tests ensuring MetricFlow time-dimension classification respects effective time-role defaulting from datatype.
converters/dbt/src/ossie_dbt/osi_to_msi.py	Switches time-dimension classification to field.is_time_dimension() (effective-role logic).
.gitignore	Ignores compiled Python artifacts (*.py[cod]).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jonmmease]

superseded by apache/ossie#113