Route DataFrame key columns through the native animation-key encoder

CHANGELOG.md

@@ -63,12 +63,17 @@ in the README).
   integers, plus finite floating arrays, including non-native-endian NumPy
   arrays. Float16/32 values widen exactly to the f64 token contract. It
   preserves the existing 64-bit identities and duplicate-row errors; mixed
-  objects, NUL-containing Python sequences, dates, and non-finite row
-  diagnostics stay on the conservative Python reference path. Highly padded
-  Python string/bytes sequences also stay there to bound fixed-width temporary
-  memory. Fixed-width NumPy strings retain their exact embedded-NUL semantics
-  natively.
-  This adds the C ABI v43 transition-key kernel.
+  objects, dates, and non-finite row diagnostics stay on the conservative
+  Python reference path. Highly padded Python string/bytes sequences also stay
+  there to bound fixed-width temporary memory. Fixed-width NumPy strings retain
+  their exact embedded-NUL semantics natively.
+  Routing follows the values rather than the container, so a key column passed
+  as a pandas Series or as homogeneous object storage — what `data=df,
+  key="id"` actually resolves to — takes the same path as an ndarray instead of
+  falling back. Only keys that *end* in NUL stay on the reference encoder,
+  where fixed-width padding would otherwise absorb them; interior NULs are
+  encoded natively.
+  This adds the C ABI v44 transition-key kernel.
 - Host theme changes made through an ancestor `data-theme` attribute now
   refresh canvas/SVG paint just like `.dark` class and inline-style changes;
   DOM chrome continues to follow the cascade automatically.

benchmarks/test_codspeed_animation.py

@@ -19,7 +19,7 @@
 
 import xy
 from xy import kernels as k
-from xy.components import _encode_transition_keys
+from xy.components import _encode_transition_keys, _fixed_transition_key_values
 
 N = 100_000
 
@@ -72,6 +72,15 @@ def payload_figures(animation_data):
 def test_animation_encode_100k_stable_keys(benchmark, animation_data) -> None:
     """Track the public list-to-native path, including homogeneous routing."""
     _x, _y, keys = animation_data
+    # Routing is the whole point of this row, and nothing about the result
+    # distinguishes the two paths — both return the same F-order planes. Prove
+    # the native kernel accepts this input before timing anything, so a
+    # layout-contract drift shows up as a red row instead of a silent ~7×
+    # regression into the reference encoder.
+    fixed = _fixed_transition_key_values(keys)
+    assert fixed is not None, "benchmark keys must route to the native encoder"
+    assert k.transition_keys_fixed(fixed, "benchmark key") is not None
+
     encoded = benchmark(_encode_transition_keys, keys, N, "benchmark key")
     assert encoded.shape == (N, 2)
     assert encoded.dtype == np.uint32

python/xy/_native.py

@@ -24,7 +24,7 @@
 
 from .config import MAX_CONTOUR_WORK, MAX_SCREEN_DIM
 
-ABI_VERSION = 43
+ABI_VERSION = 44
 
 # Rust reports invalid arguments (and, via the ffi_guard panic shield, any
 # internal panic) by returning `usize::MAX` from size-returning entry points.
@@ -971,11 +971,17 @@ def transition_keys_fixed(
     """Encode homogeneous fixed-width transition keys in one native row scan.
 
     The returned ``(N, 2)`` array is Fortran-contiguous: its two ``u32``
-    columns are the caller-allocated lo/hi planes written by Rust and can be
-    shipped independently without another full-size copy. ``None`` asks the
-    policy layer to use its scalar oracle for an invalid value (for example a
-    non-finite float or invalid Unicode scalar), preserving its precise
-    user-facing error.
+    columns are the caller-allocated lo/hi planes written by Rust, so an
+    unreordered result ships each plane without another full-size copy.
+    Reordering the rows (the line-like geometry sort, or a finite-row
+    selection) hands back C-order and restores the usual per-column copy at
+    ship time.
+
+    ``None`` asks the policy layer to use its scalar oracle for a value this
+    kernel declines (a non-finite float, an invalid Unicode scalar),
+    preserving its precise user-facing error. An invalid *argument* is a bug
+    here rather than a data property, and raises instead of degrading
+    silently into the reference path.
     """
     records = np.asarray(values)
     if records.ndim != 1 or records.dtype.hasobject:
@@ -1040,6 +1046,11 @@ def transition_keys_fixed(
         )
     if status == 3:
         raise ValueError(f"{label} produced an identity digest collision")
+    if status == 4:
+        raise RuntimeError(
+            "native transition-key encoder rejected the "
+            f"{records.dtype.str!r} layout it was handed (kind {kind}, width {width})"
+        )
     raise RuntimeError(f"native transition-key encoder returned unknown status {status}")
 
 

python/xy/components.py

@@ -3872,62 +3872,101 @@ def _transition_key_token(value: Any, index: int) -> bytes:
 _TRANSITION_KEY_SEQUENCE_MAX_PADDING_RATIO = 8
 
 
-def _fixed_transition_key_values(value: Any) -> np.ndarray | None:
-    """Return a conservative homogeneous array for the native key encoder."""
-    if type(value) is np.ndarray:
-        arr = value
-    elif isinstance(value, (list, tuple)) and value:
-        first = value[0].item() if isinstance(value[0], np.generic) else value[0]
-        item_type = type(first)
-        if item_type not in (str, bytes, bool, int, float):
+_TRANSITION_KEY_SEQUENCE_KINDS = {
+    str: {"U"},
+    bytes: {"S"},
+    bool: {"b"},
+    int: {"i", "u"},
+    float: {"f"},
+}
+
+
+def _column_ndarray(value: Any) -> np.ndarray | None:
+    """Unwrap a dtype-carrying column (pandas/polars Series) as plain NumPy.
+
+    `data=df, key="id"` resolves to a Series, not an ndarray, so without this
+    the whole native path would be unreachable from the documented idiom.
+    """
+    if isinstance(value, (str, bytes)) or not hasattr(value, "to_numpy"):
+        return None
+    try:
+        arr = value.to_numpy()
+    except (TypeError, ValueError):
+        # Extension arrays may refuse a zero-copy conversion; the reference
+        # encoder reads the same values through the object protocol anyway.
+        return None
+    return arr if type(arr) is np.ndarray else None
+
+
+def _fixed_sequence_array(sequence: Any) -> np.ndarray | None:
+    """Convert a homogeneous scalar sequence to fixed-width NumPy storage.
+
+    Returns None whenever the conversion would change an identity or cost
+    disproportionate memory, leaving the row to the Python reference encoder.
+    """
+    items = sequence.tolist() if isinstance(sequence, np.ndarray) else sequence
+    if not items:
+        return None
+    first = items[0]
+    if isinstance(first, np.generic):
+        first = first.item()
+    item_type = type(first)
+    if item_type not in _TRANSITION_KEY_SEQUENCE_KINDS:
+        return None
+    if set(map(type, items)) != {item_type}:
+        # Either genuinely mixed, or NumPy scalars needing ``.item()``. Retry
+        # through the scalar protocol, then insist on one exact builtin type:
+        # never coerce mixed bool/int or int/float keys, whose scalar tokens
+        # are deliberately type-sensitive.
+        items = [raw.item() if isinstance(raw, np.generic) else raw for raw in items]
+        if set(map(type, items)) != {item_type}:
             return None
-        total_units = 0
-        max_units = 0
-        for raw in value:
-            item = raw.item() if isinstance(raw, np.generic) else raw
-            if type(item) is not item_type:
-                # In particular, never coerce mixed bool/int or int/float
-                # keys: their scalar tokens are deliberately type-sensitive.
-                return None
-            if (item_type is str and "\x00" in item) or (item_type is bytes and b"\x00" in item):
-                # Fixed-width storage cannot distinguish padding from an
-                # explicitly trailing NUL supplied in a Python scalar.
-                return None
-            if item_type in (str, bytes):
-                units = len(item)
-                total_units += units
-                max_units = max(max_units, units)
-        if item_type in (str, bytes):
-            unit_bytes = 4 if item_type is str else 1
-            # NumPy's fixed-width sequence conversion costs N × max width.
-            # Keep a single long outlier from turning a compact Python list
-            # into a huge temporary before Rust can scan it.
-            padded_bytes = len(value) * max(max_units, 1) * unit_bytes
-            source_bytes = max(total_units * unit_bytes, 1)
-            if (
-                padded_bytes > _TRANSITION_KEY_SEQUENCE_MAX_FIXED_BYTES
-                or padded_bytes > source_bytes * _TRANSITION_KEY_SEQUENCE_MAX_PADDING_RATIO
-            ):
-                return None
-        try:
-            arr = np.asarray(value)
-        except (OverflowError, TypeError, ValueError):
+    if item_type in (str, bytes):
+        lengths = list(map(len, items))
+        unit_bytes = 4 if item_type is str else 1
+        # NumPy's fixed-width conversion costs N × longest key. Keep a single
+        # long outlier from turning a compact sequence into a huge temporary
+        # before Rust can scan it. The ratio compares padding to payload in
+        # those same fixed-width units — it is not a bound on the growth over
+        # the source objects' own footprint; the absolute cap is.
+        padded_bytes = len(items) * max(max(lengths), 1) * unit_bytes
+        source_bytes = max(sum(lengths) * unit_bytes, 1)
+        if (
+            padded_bytes > _TRANSITION_KEY_SEQUENCE_MAX_FIXED_BYTES
+            or padded_bytes > source_bytes * _TRANSITION_KEY_SEQUENCE_MAX_PADDING_RATIO
+        ):
             return None
-        expected_kinds = {
-            str: {"U"},
-            bytes: {"S"},
-            bool: {"b"},
-            int: {"i", "u"},
-            float: {"f"},
-        }
-        if arr.dtype.kind not in expected_kinds[item_type]:
-            # NumPy may coerce an integer list outside its fixed-width range
-            # to f64. That would change the type-sensitive ``i:`` token into
-            # an ``f:`` token, so leave arbitrary-width integers to Python.
+        # A key that *ends* in NUL is indistinguishable from padding once
+        # stored fixed-width, which would silently retokenize it. Interior
+        # NULs survive the round trip and stay on the native path.
+        nul = "\x00" if item_type is str else b"\x00"
+        if any(item.endswith(nul) for item in items):
             return None
-    else:
+    try:
+        arr = np.asarray(items)
+    except (OverflowError, TypeError, ValueError):
         return None
-    if arr.ndim != 1:
+    if arr.dtype.kind not in _TRANSITION_KEY_SEQUENCE_KINDS[item_type]:
+        # NumPy may coerce an integer list outside its fixed-width range to
+        # f64. That would change the type-sensitive ``i:`` token into an
+        # ``f:`` token, so leave arbitrary-width integers to Python.
+        return None
+    return arr
+
+
+def _fixed_transition_key_values(value: Any) -> np.ndarray | None:
+    """Return a conservative homogeneous array for the native key encoder."""
+    arr = value if type(value) is np.ndarray else _column_ndarray(value)
+    if arr is None:
+        if not isinstance(value, (list, tuple)):
+            return None
+        arr = _fixed_sequence_array(value)
+    elif arr.dtype.hasobject:
+        # Object storage — a pandas string column, or an explicitly
+        # object-dtype array — still reaches the encoder when every row
+        # carries the same builtin scalar type.
+        arr = _fixed_sequence_array(arr) if arr.ndim == 1 else None
+    if arr is None or arr.ndim != 1:
         return None
     if arr.dtype.kind == "U" and arr.dtype.itemsize > 0 and arr.dtype.itemsize % 4 == 0:
         return arr

scripts/abi_smoke.py

@@ -568,8 +568,44 @@ def ok(cond: bool, msg: str) -> None:
             ctypes.byref(transition_first),
             ctypes.byref(transition_index),
         )
-        == 1,
-        "transition_keys_fixed non-empty/null rejected",
+        == 4,
+        "transition_keys_fixed non-empty/null is an argument error",
+    )
+    nonfinite_records = array("d", [float("inf")])
+    nonfinite_low = array("I", [99])
+    nonfinite_high = array("I", [99])
+    transition_first = ctypes.c_size_t(size_max)
+    transition_index = ctypes.c_size_t(size_max)
+    ok(
+        lib.xy_transition_keys_fixed(
+            _ptr(nonfinite_records, ctypes.c_uint8),
+            1,
+            8,
+            5,
+            0,
+            _ptr(nonfinite_low, ctypes.c_uint32),
+            _ptr(nonfinite_high, ctypes.c_uint32),
+            ctypes.byref(transition_first),
+            ctypes.byref(transition_index),
+        )
+        == 1
+        and transition_first.value == 0,
+        "transition_keys_fixed declines non-finite data with its row",
+    )
+    ok(
+        lib.xy_transition_keys_fixed(
+            _ptr(nonfinite_records, ctypes.c_uint8),
+            1,
+            3,
+            0,
+            0,
+            _ptr(nonfinite_low, ctypes.c_uint32),
+            _ptr(nonfinite_high, ctypes.c_uint32),
+            ctypes.byref(transition_first),
+            ctypes.byref(transition_index),
+        )
+        == 4,
+        "transition_keys_fixed bad layout is an argument error",
     )
     small_unique = array("I", [99] * 2)
     ok(

spec/design/animation.md

@@ -66,17 +66,31 @@ signed or unsigned integers, and finite floating arrays take one bulk native
 path, including non-native endian NumPy arrays; f16/f32 values widen exactly to
 the f64 token contract. Its identities and first-duplicate row diagnostics are
 byte-for-byte compatible with the original Python encoder.
-Python remains the conservative reference path for mixed objects,
-NUL-containing Python string/bytes sequences, dates, and datetimes, and
-supplies the exact row error for non-finite numbers. Python sequences whose
-fixed-width conversion would exceed the byte budget or 8× padding ratio also
-stay on that path, bounding temporary memory under skewed key lengths.
-Fixed-width NumPy `U`/`S` records keep their exact trailing-padding and
-embedded-NUL semantics in the native path. This keeps the complete public
+Routing is over values, not containers: lists, tuples, NumPy arrays, and
+`to_numpy`-carrying columns (the pandas Series a `data=df, key="id"` lookup
+actually returns) all reach it, and object storage qualifies whenever every row
+holds the same builtin scalar type — otherwise the documented DataFrame idiom
+would be the one shape that never took the fast path.
+Python remains the conservative reference path for mixed objects, dates,
+datetimes, and arbitrary-width integers, and supplies the exact row error for
+non-finite and missing values. Sequences whose fixed-width conversion would
+exceed the byte budget or 8× padding ratio also stay on that path, bounding
+temporary memory under skewed key lengths.
+Fixed-width `U`/`S` records keep their exact trailing-padding and embedded-NUL
+semantics in the native path; a Python key that *ends* in NUL would be
+indistinguishable from that padding, so only those stay on the reference path —
+interior NULs survive the round trip. This keeps the complete public
 grammar—strings, finite numbers, booleans, bytes, dates, datetimes, and NumPy
 equivalents—without pushing Python object policy through the C ABI. Missing,
 unsupported, wrong-length, or duplicate values fail during figure construction.
-Line-like keys follow the same stable geometry sort as their coordinates.
+The kernel separates "declined this data, use the oracle" from "this layout is
+not in the ABI": only the first falls back, so a contract drift between the
+ctypes gate and the Rust layout check raises instead of silently costing the
+whole speedup.
+Line-like keys follow the same stable geometry sort as their coordinates; the
+encoder hands back Fortran-order planes that ship without a per-column copy,
+but that sort and any finite-row selection reorder through NumPy advanced
+indexing, which returns C-order and restores the copy at ship time.
 Errorbar point keys are role-qualified after expansion so the main segment and
 caps remain unique and stable.