fix: resolve benchmark prompt assets without asserting concrete asset types

[RapidPoseidon]

Problem

RapidataBenchmark.identifiers (and prompts / prompt_assets / tags) crashes on any benchmark whose prompts have file assets — which blocks add_model and add_prompts, since both read self.identifiers internally.

__instantiate_prompts did:

assert isinstance(prompt.prompt_asset.actual_instance, FileAssetModel)
source_url = prompt.prompt_asset.actual_instance.metadata["sourceUrl"].actual_instance
assert isinstance(source_url, SourceUrlMetadataModel)

Neither assumption holds against the current API response:

1. prompt.prompt_asset is an IAssetModel oneOf wrapper; its .actual_instance is an IAssetModel* variant (IAssetModelFileAssetModel / …MultiAssetModel / …TextAssetModel / …NullAssetModel) — never the bare FileAssetModel. → AssertionError.
2. metadata is Dict[str, IMetadataModel] and only carries a "sourceUrl" entry for URL-registered assets. File-uploaded prompts only have originalFilename / imageDimension, so metadata["sourceUrl"] would KeyError.
3. The metadata variant is IMetadataModelSourceUrlMetadataModel, not the bare SourceUrlMetadataModel, so the second assert is wrong too.

Reproduced live against a real file-asset benchmark (499 prompts): benchmark.identifiers raised AssertionError before this change.

Fix

Resolve the source URL defensively instead of asserting concrete types, mirroring the existing pattern in RapidataFlowItem._extract_asset_key():

@staticmethod
def __source_url(asset):
    instance = getattr(asset, "actual_instance", None)
    metadata = getattr(instance, "metadata", None)
    if not metadata:
        return None
    source_url = metadata.get("sourceUrl")
    actual = getattr(source_url, "actual_instance", None)
    return getattr(actual, "url", None)

prompt_assets now yields the source URL when present, else None for file-uploaded assets that have no public URL in the response — the old code could never have returned a URL for those anyway; it crashed.

Verification

• Live against the real benchmark (read-only, creates nothing): identifiers, prompts, prompt_assets, tags all populate (499 each) with no exception; add_model's identifier-membership check passes.
• uv run black src/rapidata/rapidata_client → unchanged.
• uv run pyright src/rapidata/rapidata_client/benchmark/rapidata_benchmark.py → 0 errors, 0 warnings.

Notes

• No version bump — versioning/publishing is CI-automated (release_and_publish.yml).
• No test added — the repo has no test framework yet; a regression test for this (IAssetModel* wrapper + missing sourceUrl) would be worth adding once test infra exists.
• Discovered while adding participants to a mock benchmark in Image-Editing-Benchmarks (RapidataAI/Image-Editing-Benchmarks#6), which currently ships a consumer-side cache-injection workaround that can be removed once this releases.

🔗 Session: ${POSEIDON_SESSION_URL:-$HOSTNAME}

🤖 Generated with Claude Code

[claude]

Code Review

Overview

This PR fixes a real crash in RapidataBenchmark.__instantiate_prompts when processing benchmarks whose prompts have file assets. The old code made two incorrect assumptions about the auto-generated API client's response structure — asserting bare FileAssetModel/SourceUrlMetadataModel types that are actually wrapped in IAssetModel*/IMetadataModel* oneOf variants — causing an AssertionError on every file-asset benchmark and blocking add_model/add_prompts as a result. The fix is targeted and correct.

---

Correctness

• The root cause is accurately diagnosed: the OpenAPI client wraps concrete types in oneOf holder objects (IAssetModel), so the old assert isinstance(..., FileAssetModel) could never pass.
• Passing None to __source_url is handled gracefully — getattr(None, "actual_instance", None) returns None, so the explicit if prompt.prompt_asset is None branch is correctly collapsed into the single call.
• The metadata.get("sourceUrl") call is safe after the if not metadata guard.
• Returning None for file-uploaded assets (which have no sourceUrl in the response) preserves the pre-existing semantics — the old code would have crashed before ever returning anything for those cases anyway.

Code Quality

Minor style concern — docstring length on a private method. The __source_url docstring is 6 prose lines explaining the oneOf wrapper structure. CLAUDE.md asks to keep comments to a minimum and avoid multi-line blocks unless the why is non-obvious. Here the why is genuinely non-obvious (a subtle API contract), so some comment is warranted, but the current block is verbose for a private static helper. Consider condensing to one or two lines:

# IAssetModel is a oneOf wrapper; only URL-registered assets carry a "sourceUrl"
# metadata entry — navigate defensively rather than asserting concrete types.

That captures the essential constraint without repeating the full type hierarchy already visible in the surrounding code.

TYPE_CHECKING Import

IAssetModel is correctly placed under TYPE_CHECKING, consistent with the project convention from CLAUDE.md.

Tooling

PR description confirms:

• uv run black — no changes
• uv run pyright src/rapidata/rapidata_client/benchmark/rapidata_benchmark.py — 0 errors, 0 warnings

Potential Follow-ups (not blocking)

1. Multi-asset / text-asset prompt types — IAssetModelMultiAssetModel and IAssetModelTextAssetModel variants presumably also lack a sourceUrl; they will silently return None here. That is likely correct behavior, but worth a note so callers of prompt_assets know a None entry can mean "file-uploaded or non-URL asset type", not just "no asset".
2. Test coverage — the PR correctly notes the repo has no test framework yet. A regression test covering the IAssetModel oneOf wrapper + missing sourceUrl path would be a high-value first test once infra exists.

---

Summary

Clean, well-scoped fix. The defensive getattr navigation mirrors the existing pattern in RapidataFlowItem._extract_asset_key and is the right approach for working with auto-generated oneOf wrappers. The only non-blocking suggestion is trimming the docstring to match the project's comment style. Approved.

[claude]

Review of PR #625 — fix: resolve benchmark prompt assets without asserting concrete asset types

This PR correctly fixes a real crash: the old code asserted isinstance(prompt.prompt_asset.actual_instance, FileAssetModel) but the API now returns an IAssetModel oneOf wrapper whose actual_instance is never the bare FileAssetModel. The defensive getattr navigation in __source_url is the right structural fix and mirrors the existing _extract_asset_key pattern. Verified cleanly against the type hierarchy.

Two actionable issues and two design notes below.

---

1. Redundant quoted annotation (CLAUDE.md violation) — rapidata_benchmark.py:56

def __source_url(asset: "IAssetModel | None") -> str | None:

The file already has from __future__ import annotations at line 1, which makes every annotation a lazy string automatically. The explicit quotes around "IAssetModel | None" are redundant and directly contradict the project convention:

"use the from __future__ import annotations statement and TYPE_CHECKING to check the types — that way you can eliminate the quotation marks around the types"

Fix: drop the quotes — asset: IAssetModel | None.

---

2. O(n²) list membership in add_model — rapidata_benchmark.py:590

if not all(identifier in self.identifiers for identifier in identifiers):

self.identifiers returns a list[str], so each in test is O(n). For 10 000 registered identifiers checked against 10 000 model outputs this is 100 M operations. The same hazard was already identified and fixed in add_prompts at line 358, which includes an explanatory comment:

# Snapshot once: `self.identifiers` is a property whose getter re-fetches
# over HTTP while the cache is empty...
existing_identifiers = set(self.identifiers)

add_model should carry the same fix:

registered = set(self.identifiers)
if not all(identifier in registered for identifier in identifiers):

This is pre-existing, but since the PR touches adjacent code it's a good opportunity to close the inconsistency.

---

3. Double-underscore name mangling on a @staticmethod (design note)

__source_url is name-mangled to _RapidataBenchmark__source_url, making it completely unreachable from outside the class — including from RapidataFlowItem._extract_asset_key, which solves the same problem on a raw dict. The PR description even calls out the parallel with _extract_asset_key as a reference pattern.

Since the method has no dependency on the class at all, a single underscore (_source_url) or a module-level helper would keep the door open for future consolidation without paying the cognitive overhead of mangling. Not a correctness issue today, but name-mangling a static helper that is already acknowledged to duplicate an existing pattern compounds the duplication rather than reducing it.

---

4. Silent None replaces loud assertion on unexpected API shapes (informational)

The old assert isinstance(...) would have failed loudly if the API shape changed in an unexpected direction. The new getattr chains silently return None for any deviation — a renamed key (sourceUrl → source_url), a missing actual_instance, or a future new variant would all produce None in prompt_assets with no log message or exception surfaced to the caller.

This is a deliberate trade-off (crashing is worse than returning None for file-uploaded assets with no public URL), but it would be worth adding a single logger.debug or logger.warning inside __source_url for the case where an asset exists but no URL is extractable — that would make future regressions diagnosable without reintroducing hard failures.

---

Overall: the core fix is correct and the PR description is thorough. Items 1 and 2 are the only changes I'd ask for before merging.

[LinoGiger]

had to fix it in a different way: #634

[RapidPoseidon]

Thanks @LinoGiger — closing this in favor of #634, which is merged and is the better fix: falling back to originalFilename (so prompt_assets returns a value for file-uploaded assets instead of my None) and normalizing the cached value so it stays consistent across a re-fetch are both improvements over what I had here. Same root cause (the IAssetModel/IMetadataModel oneOf wrappers vs. the bare FileAssetModel/SourceUrlMetadataModel), better handling.

For traceability: this surfaced while adding participants to a benchmark with image prompts (add_model → identifiers crashed). The downstream repo (RapidataAI/Image-Editing-Benchmarks#6, merged) carries a temporary cache-injection workaround; I'll drop it once a release containing #634 is published (not tagged yet as of this writing).