Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d6ecb83
wip - Finished @forbid_fields_if, only to realize it's not right either
vcschapp Oct 9, 2025
4a6c675
wip - CHECKPOINT - fix bug in divisions model and migrate @forbid_if,…
vcschapp Oct 9, 2025
94f9438
wip - CHECKPOINT - re-homed `exactly_one_of` as `radio_group`, test d…
vcschapp Oct 10, 2025
4f1d2d3
wip - PASSING - finalize validation -> system (except ExtensibleBaseM…
vcschapp Oct 10, 2025
c809b4f
wip - Record FIXME for Pydantic warning
vcschapp Oct 10, 2025
999e4c0
wip - ensure JSON Schema / Pydantic validation parity on required/not…
vcschapp Oct 10, 2025
ca1c80d
wip - SEMI-GOOD | all unit tests done, system is passing pytest & ruf…
vcschapp Oct 11, 2025
67ab3ff
wip [PASSING] - fix weird doctest regressions
vcschapp Oct 14, 2025
babec58
wip - Delete unused core primitives module
vcschapp Oct 14, 2025
dea80b4
wip: [MESS] middle of Feature factoring
vcschapp Oct 15, 2025
aaf0646
wip - [CHECKPOINT] doctests passing for Omitable and Feature
vcschapp Oct 15, 2025
dcb75ad
wip - [BROKEN] - Closing innings of getting mixed-level feature JSON …
vcschapp Oct 16, 2025
af29a35
wip - [CHECKPOINT] - Feature tests finally passing but make check sti…
vcschapp Oct 16, 2025
51d7741
wip - Add `make coverage` with HTML report
vcschapp Oct 16, 2025
51f74dd
wip - Minor
vcschapp Oct 16, 2025
7ef9d3c
wip - PASSING - Feature complete (pun intended) and make check is pas…
vcschapp Oct 16, 2025
adb57c6
wip - [PASSING] - Migrate ref down to its new home in system
vcschapp Oct 16, 2025
b6dda78
wip - RC - PASSING - Port core Feature to use system feature, FIX BUG…
vcschapp Oct 16, 2025
498d3f7
wip - re-baseline due to bbox fix
vcschapp Oct 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ test-all: uv-sync
test: uv-sync
@uv run pytest packages/ -x

coverage: uv-sync
@uv run pytest packages/ --cov overture.schema --cov-report=term --cov-report=html && open htmlcov/index.html

docformat:
@find packages/*/src -name "*.py" -type f -not -name "__*" \
| xargs uv run pydocstyle --convention=numpy --add-ignore=D105
Expand Down
2 changes: 1 addition & 1 deletion README.pydantic.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ This workspace contains the following packages:
convenient usage
- **`overture-schema-core`** - Base classes, geometry models, and common structures
shared across all themes
- **`overture-schema-validation`** - Validation system with constraints and mixins
- **`overture-schema-system`** - Foundational system of primitivef types and constraints

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **`overture-schema-system`** - Foundational system of primitivef types and constraints
- **`overture-schema-system`** - Foundational system of primitive types and constraints


### Theme Packages

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,15 @@
from pydantic import BaseModel, ConfigDict, Field

from overture.schema.core import (
Feature,
OvertureFeature,
)
from overture.schema.core.types import CountryCodeAlpha2
from overture.schema.system.model_constraint import no_extra_fields
from overture.schema.system.primitive import (
Geometry,
GeometryType,
GeometryTypeConstraint,
)
from overture.schema.system.string import StrippedString
from overture.schema.system.string import CountryCodeAlpha2, StrippedString


@no_extra_fields
Expand All @@ -35,7 +34,7 @@ class AddressLevel(BaseModel):
] = None


class Address(Feature[Literal["addresses"], Literal["address"]]):
class Address(OvertureFeature[Literal["addresses"], Literal["address"]]):
"""Addresses are geographic points used for locating businesses and individuals. The
rules, fields, and fieldnames of an address can vary extensively between locations.
We use a simplified schema to capture worldwide address points. This initial schema
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,9 @@
},
"additionalProperties": false,
"description": "Addresses are geographic points used for locating businesses and individuals. The\nrules, fields, and fieldnames of an address can vary extensively between locations.\nWe use a simplified schema to capture worldwide address points. This initial schema\nis largely based on the OpenAddresses (www.openaddresses.io) project.\n\nThe address schema allows up to 5 \"admin levels\". Rather than have field names that\napply across all countries, we provide an array called \"address_levels\" containing\nthe necessary administrative levels for an address.",
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
}
},
Comment on lines -78 to -82

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one of the subtle 🐞 bugs addressed: patternProperties shouldn't exist at the top level because that would allow extra properties to be put into the root of the GeoJSON document, like this:

{
  "type": "Feature",
  "geometry": {},
  "properties": {},
  "ext_foo": "bar",
  "ext_bar": "baz"
}

The extension properties would then get lost by the model validator that imports the GeoJSON into the Pydantic model class instance.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm, so it wasn't getting pivoted into properties correctly. Good catch.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was getting cloned into properties, but was somehow also getting replicated at the top level.

"properties": {
"bbox": {
"description": "An optional bounding box for the feature",
"items": {
"type": "number"
},
Expand Down Expand Up @@ -129,9 +125,16 @@
},
"properties": {
"additionalProperties": false,
"not": {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes another subtle 🐞 bug: if these properties were allowed in under "properties", then you'd be open to this scenario in a GeoJSON input:

{
  "type": "Feature",
  "id": "outer-id",
  "bbox": [0, 0, 0, 0],
  "geometry": {},
  "properties": {
    "id": "inner-id",
    "bbox": [1, 1, 1, 1],
    "geometry": "anything"
  }
}

The model validator would then copy everything from "properties" up to the root (overwriting the values in the root).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not.required achieve this? It seems more like it would say that those fields aren't required, which was already the case. additionalProperties: false will prevent it, but is contingent on the shape of properties filtering out references to id, bbox, geometry.

@vcschapp Victor Schappert (vcschapp) Oct 20, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required is a funny beast.

The positive version means "it must be the case that it is there" and the negative version means "it must not be the case that it is there".

So if you check out this mini-schema at https://www.jsonschemavalidator.net/...

{
  "type": "object",
  "not": { "required": ["foo" ] },
  "required": ["bar"]
}

You find that this passes: {"bar": 42}, but this fails: {"foo": "anything", "bar": 42}.


Edit: Fixed an example that wrongly ended in a bracket ] instead of a brace }.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL, wow. That's worth capturing (if it's not already) in the code that produces expressions like that.

"required": [
"id",
"bbox",
"geometry"
]
},
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
"description": "Additional top-level properties are allowed if prefixed by `ext_`.\n\nThis feature is a on a deprecation path and will be removed once the schema is\nfully migrated to Pydantic."
}
},
"properties": {
Expand Down Expand Up @@ -221,19 +224,18 @@
"type",
"version"
],
"type": "object",
"unevaluatedProperties": false

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant - because "additionalProperties": false is stricter...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by "stricter"?

IIRC, unevaluatedProperties applies to sub-schemas (and structures introduced through allOf, etc.), which is probably what we want at the OvertureFeature level (to prevent people from accidentally making their schemas more open than intended, as is the case in a few parts of our current JSON Schema).

@vcschapp Victor Schappert (vcschapp) Oct 20, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean by stricter is that "additionalProperties": false will prohibit any properties from being included if they are not either explicitly listed in properties or covered by an allowed pattern in patternProperties. So the fact that it won't traverse sub-schemas introduced through aggregators allOf, anyOf, etc.) makes it stricter.

e.g. consider this JSON Schema:

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
     "foo": { "type": "integer" } 
  },
  "patternProperties": {
    "bar.*": { }
  },
  "allOf": [{
    "properties": {
      "baz": { }
    }
  }]
}

The above will work for {"foo":42,"bark": "anything"} but not {"foo":42,"bark": "anything", "baz":"anything else"}.

The reason this makes sense to me in the Pydantic context is that Pydantic models always have enumerated properties in the properties section. The one exception so far is ext_*, which is covered by patternProperties. All of our other applicable constraints don't add allowed properties, they just constrain the values the existing properties are allowed to have.


Edit: Added a missing not that was super confusing.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aie, that's confusing.

Given the behavior you're showing, we might do well to avoid ever creating aggregators that introduce sub-schemas.

"type": "object"
},
"type": {
"const": "Feature",
"type": "string"
}
},
"required": [
"type",
"id",
"geometry",
"properties",
"type"
"properties"
],
"title": "address",
"type": "object"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

from pydantic import BaseModel, Field, HttpUrl

from overture.schema.core.types import CountryCodeAlpha2
from overture.schema.system.model_constraint import no_extra_fields
from overture.schema.system.string import CountryCodeAlpha2

from .enums import BuildSource, UpdateType
from .types import LicenseShortname
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

from overture.schema.base.types import Depth
from overture.schema.core import (
Feature,
OvertureFeature,
Comment thread
vcschapp marked this conversation as resolved.
)
from overture.schema.core.models import CartographicallyHinted
from overture.schema.system.primitive import (
Expand All @@ -17,7 +17,7 @@


class Bathymetry(
Feature[Literal["base"], Literal["bathymetry"]], CartographicallyHinted
OvertureFeature[Literal["base"], Literal["bathymetry"]], CartographicallyHinted
):
"""Topographic representation of an underwater area, such as a part of the ocean
floor."""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from overture.schema.base.models import SourcedFromOpenStreetMap
from overture.schema.base.types import Height
from overture.schema.core import (
Feature,
OvertureFeature,
)
from overture.schema.core.models import Named, Stacked
from overture.schema.system.primitive import (
Expand All @@ -24,7 +24,7 @@


class Infrastructure(
Feature[Literal["base"], Literal["infrastructure"]],
OvertureFeature[Literal["base"], Literal["infrastructure"]],
Named,
Stacked,
SourcedFromOpenStreetMap,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from overture.schema.base.models import SourcedFromOpenStreetMap
from overture.schema.base.types import Elevation
from overture.schema.core import (
Feature,
OvertureFeature,
)
from overture.schema.core.models import Named, Stacked
from overture.schema.system.primitive import (
Expand All @@ -21,7 +21,10 @@


class Land(
Feature[Literal["base"], Literal["land"]], Named, Stacked, SourcedFromOpenStreetMap
OvertureFeature[Literal["base"], Literal["land"]],
Named,
Stacked,
SourcedFromOpenStreetMap,
):
"""Physical representations of land surfaces.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

from overture.schema.base.land_cover.enums import LandCoverSubtype
from overture.schema.core import (
Feature,
OvertureFeature,
)
from overture.schema.core.models import CartographicallyHinted
from overture.schema.system.primitive import (
Expand All @@ -17,7 +17,7 @@


class LandCover(
Feature[Literal["base"], Literal["land_cover"]], CartographicallyHinted
OvertureFeature[Literal["base"], Literal["land_cover"]], CartographicallyHinted
):
"""Representation of the Earth's natural surfaces."""

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from overture.schema.base.models import SourcedFromOpenStreetMap
from overture.schema.base.types import Elevation
from overture.schema.core import (
Feature,
OvertureFeature,
)
from overture.schema.core.models import Named, Stacked
from overture.schema.system.primitive import (
Expand All @@ -21,7 +21,7 @@


class LandUse(
Feature[Literal["base"], Literal["land_use"]],
OvertureFeature[Literal["base"], Literal["land_use"]],
Named,
Stacked,
SourcedFromOpenStreetMap,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from overture.schema.base.models import SourcedFromOpenStreetMap
from overture.schema.base.water.enums import WaterClass, WaterSubtype
from overture.schema.core import (
Feature,
OvertureFeature,
)
from overture.schema.core.models import Named, Stacked
from overture.schema.system.primitive import (
Expand All @@ -18,7 +18,10 @@


class Water(
Feature[Literal["base"], Literal["water"]], Stacked, Named, SourcedFromOpenStreetMap
OvertureFeature[Literal["base"], Literal["water"]],
Stacked,
Named,
SourcedFromOpenStreetMap,
):
"""Physical representations of inland and ocean marine surfaces.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,13 +94,9 @@
},
"additionalProperties": false,
"description": "Topographic representation of an underwater area, such as a part of the ocean\nfloor.",
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
}
},
"properties": {
"bbox": {
"description": "An optional bounding box for the feature",
"items": {
"type": "number"
},
Expand Down Expand Up @@ -200,9 +196,16 @@
},
"properties": {
"additionalProperties": false,
"not": {
"required": [
"id",
"bbox",
"geometry"
]
},
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
"description": "Additional top-level properties are allowed if prefixed by `ext_`.\n\nThis feature is a on a deprecation path and will be removed once the schema is\nfully migrated to Pydantic."
}
},
"properties": {
Expand Down Expand Up @@ -251,19 +254,18 @@
"version",
"depth"
],
"type": "object",
"unevaluatedProperties": false
"type": "object"
},
"type": {
"const": "Feature",
"type": "string"
}
},
"required": [
"type",
"id",
"geometry",
"properties",
"type"
"properties"
],
"title": "bathymetry",
"type": "object"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -432,13 +432,9 @@
},
"additionalProperties": false,
"description": "Various features from OpenStreetMap such as bridges, airport runways, aerialways,\nor communication towers and lines.",
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
}
},
"properties": {
"bbox": {
"description": "An optional bounding box for the feature",
"items": {
"type": "number"
},
Expand Down Expand Up @@ -598,9 +594,16 @@
},
"properties": {
"additionalProperties": false,
"not": {
"required": [
"id",
"bbox",
"geometry"
]
},
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
"description": "Additional top-level properties are allowed if prefixed by `ext_`.\n\nThis feature is a on a deprecation path and will be removed once the schema is\nfully migrated to Pydantic."
}
},
"properties": {
Expand Down Expand Up @@ -676,19 +679,18 @@
"class",
"subtype"
],
"type": "object",
"unevaluatedProperties": false
"type": "object"
},
"type": {
"const": "Feature",
"type": "string"
}
},
"required": [
"type",
"id",
"geometry",
"properties",
"type"
"properties"
],
"title": "Infrastructure Schema",
"type": "object"
Expand Down
22 changes: 12 additions & 10 deletions packages/overture-schema-base-theme/tests/land_baseline_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -306,13 +306,9 @@
},
"additionalProperties": false,
"description": "Physical representations of land surfaces.\n\nGlobal land derived from the inverse of OSM Coastlines. Translates `natural` tags from OpenStreetMap.",
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
}
},
"properties": {
"bbox": {
"description": "An optional bounding box for the feature",
"items": {
"type": "number"
},
Expand Down Expand Up @@ -472,9 +468,16 @@
},
"properties": {
"additionalProperties": false,
"not": {
"required": [
"id",
"bbox",
"geometry"
]
},
"patternProperties": {
"^ext_.*$": {
"description": "Additional top-level properties must be prefixed with `ext_`."
"description": "Additional top-level properties are allowed if prefixed by `ext_`.\n\nThis feature is a on a deprecation path and will be removed once the schema is\nfully migrated to Pydantic."
}
},
"properties": {
Expand Down Expand Up @@ -551,19 +554,18 @@
"type",
"version"
],
"type": "object",
"unevaluatedProperties": false
"type": "object"
},
"type": {
"const": "Feature",
"type": "string"
}
},
"required": [
"type",
"id",
"geometry",
"properties",
"type"
"properties"
],
"title": "land",
"type": "object"
Expand Down
Loading