Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,21 @@ class Check:
and report grouping), not how to access the data. The expression in
Comment thread
vcschapp marked this conversation as resolved.
`expr` already encodes the access pattern.

`read_columns` names every top-level schema column the expression
dereferences -- one for a plain field check, several for a model-level
check that spans columns, plus any discriminator a variant gate reads.
`validate_model` drops a check when any column it reads is skipped or
structurally absent, so an unresolvable `F.col()` never reaches Spark;
it also treats these as the columns a check can be suppressed by name.
`expr` and `read_columns` are two views of one computation, and each is
a "column" in a different sense. `read_columns` are real columns of the
underlying schema model -- the top-level columns the check must read to
evaluate. There is always at least one; a model-level constraint that
spans fields names several, plus any discriminator a variant gate reads.
`expr` is a *virtual column*: it is not a column of the schema model but
one synthesized by the generated validation machinery to hold the
composed expression the Spark engine evaluates. The two travel together
because the builder knows the read-set as it composes `expr`; recording
it is surer than recovering it from the finished `Column`.

`validate_model` drops a check when any column in `read_columns` is
skipped or structurally absent, so an unresolvable `F.col()` never
reaches Spark; it also treats these as the columns a check can be
suppressed by name.
"""

field: str
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@
"MapPath",
"MapProjection",
"MapSegment",
"PathSegment",
"ScalarPath",
"StructSegment",
"coerce",
Expand Down Expand Up @@ -88,9 +87,6 @@ class MapSegment:
projection: MapProjection


PathSegment: TypeAlias = StructSegment | ArraySegment


@dataclass(frozen=True, slots=True)
class ScalarPath:
"""Locate a non-iterated value in a row."""
Expand All @@ -116,7 +112,7 @@ class ArrayPath:
Invariant: `segments` contains at least one `ArraySegment`.
"""

segments: tuple[PathSegment, ...]
segments: tuple[StructSegment | ArraySegment, ...]

def __post_init__(self) -> None:
if not any(isinstance(s, ArraySegment) for s in self.segments):
Expand Down Expand Up @@ -334,14 +330,14 @@ def __str__(self) -> str:


# The element type of any `FieldPath.segments`, across all three variants.
# Broader than `PathSegment` (array/scalar paths only): a `MapPath` adds a
# trailing `MapSegment`. Consumers that walk an arbitrary `FieldPath`'s
# segments -- rather than a statically known `ArrayPath` -- annotate with
# this so a `MapSegment` is not a type error.
# Broader than an `ArrayPath`'s `StructSegment | ArraySegment`: a `MapPath`
# adds a trailing `MapSegment`. Consumers that walk an arbitrary
# `FieldPath`'s segments -- rather than a statically known `ArrayPath` --
# annotate with this so a `MapSegment` is not a type error.
FieldSegment: TypeAlias = StructSegment | ArraySegment | MapSegment


def _segment_str(seg: PathSegment) -> str:
def _segment_str(seg: StructSegment | ArraySegment) -> str:
if isinstance(seg, ArraySegment):
return seg.name + "[]" * seg.iter_count
return seg.name
Expand Down