Skip to content

[ENHANCEMENT](pyspark) Generalize field_path to arbitrary map/array nesting - #572

Merged
Victor Schappert (vcschapp) merged 4 commits into
mainfrom
generalize-field-path
Jul 23, 2026
Merged

[ENHANCEMENT](pyspark) Generalize field_path to arbitrary map/array nesting#572
Victor Schappert (vcschapp) merged 4 commits into
mainfrom
generalize-field-path

Conversation

@sethfitz

@sethfitz Seth Fitzsimmons (sethfitz) commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Implements #570 — generalize field_path to arbitrary map/array nesting.

What changed

Collapses field_path's three-class taxonomy (ScalarPath/ArrayPath/MapPath) into Direct + Iterated, extends the grammar to interleaved map/array markers, and replaces the four array/map wrap functions with one _wrap_in_iteration fold over RenderFrames. A container nested directly inside another with no field name between them is an anonymous iterating segment, so list[list[X]], dict[K, list], list[dict], nested maps, and maps reached through arrays are now representable and render working validation end-to-end. Adds the flattening runtime pair nested_map_{keys,values}_check.

Notes

The generated-conformance-test path now handles mixed map/array shapes: a field check on a map value/key reached through array iteration (items[].tags{value}) descends the array and mutates the right map side, ground-truthed against real Spark. The remaining mixed shapes with no in-place mutation (subs{value}[], items[].configs{value}) are loud capability gates, not silent misroutes.

Closes #570

@github-actions

github-actions Bot commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown

🗺️ Schema reference docs preview is live!
🌍 Preview https://staging.overturemaps.org/schema/pr/572/schema/index.html
🕐 Updated Jul 23, 2026 16:28 UTC
📝 Commit 394553a
🔧 env SCHEMA_PREVIEW true

Note

♻️ This preview updates automatically with each push to this PR.

@sethfitz
Seth Fitzsimmons (sethfitz) marked this pull request as ready for review July 20, 2026 21:26
@sethfitz Seth Fitzsimmons (sethfitz) mentioned this pull request Jul 20, 2026
Merged
Base automatically changed from pyspark-expression-codegen to main July 21, 2026 17:31
@sethfitz
feat(pyspark): generalize field_path to arbitrary map/array nesting
9fdb191 
Collapse field_path's ScalarPath/ArrayPath/MapPath taxonomy into Direct
+ Iterated and extend the grammar to interleaved map/array markers. A
container nested directly inside another with no field name between
(list[list[X]], dict[K, list[X]], list[dict], nested maps, a map reached
through an array) becomes an anonymous iterating segment, so every such
shape is now representable and renders, validates, and mutates end to
end.

Replace the four array/map wrap functions with one _wrap_in_iteration
fold over RenderFrames, and add the flattening runtime pair
nested_map_{keys,values}_check. Remove the union-under-multi-list and
newtype-under-list guards -- those shapes render correctly now -- and
add a guard for a union variant field reached through iteration past its
discriminator element, where the ElementGuard would gate the wrong
element.

The generated-conformance-test path handles the new shapes: set_at_path
learns the {value}/{key} grammar and descends trailing containers; model
mutations gain a composite element_path for map-then-array and
array-then-map nesting; shapes with no in-place mutation are loud
capability gates, not silent misroutes.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz
feat(pyspark): validate and mutate struct-nested model constraints
2b97f1f 
A model-level constraint declared on a submodel reached through a plain
struct field -- @require_any_of on Details where Feature.details:
Details -- now anchors at the struct prefix instead of collapsing to the
row root.  _model_constraint_target is the identity on the prefix; the
renderer qualifies every field and condition reference with that prefix
(F.col("details.foo")) and gates an optional struct on presence
(F.when(F.col("details").isNotNull(), ...)). ModelCheck.read_columns
reads the single top-level struct column.

_guard_struct_nested_anchor narrows to struct-nested discriminated
unions only: their variant ColumnGuards render the discriminator as a
top-level column, which a struct-qualified path cannot express. Plain
struct-nested model constraints are now supported and no longer raise.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz
docs(pyspark): document the Direct/Iterated taxonomy and iteration fold
baab128 
Rewrite the Check Builder section of the codegen design doc for the
current Direct/Iterated FieldPath taxonomy: anonymous segments, the
single _wrap_in_iteration fold, and the mixed map/array nesting cases
(dict[K, list], list[dict], nested maps, map-in-array), replacing the
retired ScalarPath/ArrayPath/MapPath description. Disambiguate
iter_frames (named-only structural folding) from the renderer's
per-segment _render_frames walk.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz
test(pyspark): batch per-test DataFrames into one collect
394553a 
test_column_patterns.py and test_constraint_expressions.py each built a
DataFrame and collected once per test -- ~200 Spark jobs between them.
Both now use a case registry: each _Case carries its input column, the
check over it, and a predicate on the result; a module-scoped fixture
packs every case's input into one wide single-row DataFrame, applies
every check in a single select, and collects once. This is the
batch-once pattern the generated conformance harness uses.

Every prior assertion is preserved. Timings (remaining floor is Spark
session startup):

  test_column_patterns.py         6.4s -> 4.0s   (33 -> 37 cases)
  test_constraint_expressions.py  16.9s -> 5.1s  (164 -> 165 cases)

The extra cases come from splitting valid/invalid two-row tests into
explicit per-input cases; no coverage dropped.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@vcschapp
Victor Schappert (vcschapp) merged commit 2981557 into main Jul 23, 2026
19 checks passed
@vcschapp
Victor Schappert (vcschapp) deleted the generalize-field-path branch July 23, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vcschapp Victor Schappert (vcschapp) vcschapp approved these changes
@RoelBollens-TomTom Roel Bollens (RoelBollens-TomTom) RoelBollens-TomTom approved these changes
@lowlydba John McCall (lowlydba) Awaiting requested review from lowlydba

Assignees

No one assigned

Labels

change type - cosmetic 🌹 Cosmetic change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Generalize field_path to arbitrary map/array nesting

3 participants

@sethfitz @vcschapp @RoelBollens-TomTom