demo(ca-governance): golden-query-via-workflow vs. normal agentic CA

[haiyuan-eng-google]

What

A BigQuery Conversational Analytics demo with a governance dial, built on the model-authored-workflow engine from this branch. It shows how to restrict CA to governed (golden / verified) queries structurally — not with a prompt — while keeping a normal agentic answer one dial-turn away.

Self-contained sample under contributing/samples/workflows/authored_workflow_ca_governance_demo/. No changes to src/ or sibling samples — it reuses the committed authoring.py engine.

The idea

The engine's CapabilityRegistry is the allow-list: a WorkflowSpec may only compose registered capabilities, and WorkflowSpecValidator rejects any plan referencing one that is not. So governance is a registry composition, enforced at validation:

STRICT (golden) : match_verified_query · run_frozen_query · summarize · refuse
FLEXIBLE        : … + nl2sql · dry_run · run_adhoc · freeze_verified

One agent, two surfaces: a data question is matched against the verified-query pool; a hit is answered by a frozen, auditable workflow running approved SQL on real BigQuery (thelook_ecommerce); a miss refuses under STRICT or falls through to a normal agentic agent (free-form Agent + query_thelook tool) under OPEN.

Beats (see README)

1. show modes registry diff — governance is a one-capability difference, not a prompt.
2. adversarial: just write SQL — an nl2sql plan is rejected at validation under STRICT (the same plan validates under FLEXIBLE). You can't prompt your way out.
3. revenue by country (strict) — governed hit: frozen approved SQL on real BigQuery, 0 model-drafted SQL.
4. churn cohorts (strict) — refused, outside the governed set, 0 queries.
5. churn cohorts (open mode) — same question → normal agentic answer (not a frozen workflow — the trade-off).

FLEXIBLE adds the constrained-yet-flexible middle ground: gated nl2sql fallback that promotes the result into the governed pool (assisted authoring).

Verification

• Live: all five beats run end-to-end with Gemini (Vertex global) + real BigQuery (engine-labeled bigquery).
• CI-safe: test_ca_governance_demo.py — 9 tests, no LLM / no BigQuery (governance proofs are deterministic). Run: pytest contributing/samples/workflows/authored_workflow_ca_governance_demo/test_ca_governance_demo.py -q.
• Headless driver: governance_demo.py runs the same root_agent through the beats — a live-demo backstop.
• Without credentials it degrades to a deterministic mock warehouse (engine-labeled), so it runs anywhere.

Notes

• The verified-query matcher is deterministic keyword overlap (auditable); production would use the dataset's semantic model/graph + embeddings — the governance mechanism is unchanged by that swap.
• Stacked on spike/dynamic-supervisor-concurrency because it imports the committed authoring engine.

[caohy1988]

This collapses flexible into open, and the runtime path below always uses golden_registry() + author_golden_plan(). A live prompt with (flexible) therefore never exercises the author_flexible_plan() path described in the README/NARRATIVE; on a miss it falls straight to the normal agentic fallback. To make the governance dial real, keep flexible as its own mode, select flexible_registry() + author_flexible_plan(), and reserve open for the free-form agentic fallback.

[haiyuan-eng-google]

Fixed in b256f9c. _mode_from now returns three distinct modes; FLEXIBLE selects flexible_registry() + author_flexible_plan(), OPEN is reserved for the agentic fallback, STRICT refuses on a miss. Added test_mode_routing_is_three_distinct_modes. Live-verified: a (flexible) miss authors the gated nl2sql plan, runs it, and promotes.

[caohy1988]

Sql only accepts sql, so the real nl2sql output cannot carry the original question forward. Even if the model returns a question field, Pydantic will drop it, _dry_run() will set question to "", and freeze_verified will promote an empty-question record. The tests hide this because the stub returns a raw dict with question. Add question: str = "" to Sql (and make the instruction return it), or otherwise bind the task question into the dry-run/run/freeze path, then assert the promoted record keeps the original question.

[haiyuan-eng-google]

Fixed in b256f9c. Added question: str = "" to Sql and updated the nl2sql instruction to copy the question verbatim, so it survives into dry_run/run/freeze. test_flexible_falls_back_validates_and_promotes_with_question now asserts the promoted record keeps the original question. Live run confirmed the promoted query_id is derived from the real question.

[caohy1988]

Right now dry_run is observational, not a gate: run_adhoc and freeze_verified run regardless of check.valid or check.error. In real BigQuery an invalid generated query would return an error from run_query, but this plan still reaches freeze_verified and promotes the SQL. Since the demo story says FLEXIBLE validates before running/promoting, add a branch on check.valid (or make run_adhoc/freeze_verified refuse invalid inputs) and add a test where nl2sql returns invalid SQL and no query is run or frozen.

[haiyuan-eng-google]

Fixed in b256f9c. The dry-run is now a real gate: author_flexible_plan() branches on check.valid — valid -> run_adhoc + freeze + summarize; invalid -> new reject_invalid leaf (nothing run, nothing promoted). Added test_flexible_gate_rejects_invalid_sql_no_run_no_freeze (asserts no adhoc/freeze state and the pool is unchanged).

[caohy1988]

run_query is documented as read-only, and the agentic tool instruction asks for read-only SELECTs, but the tool does not enforce that before calling BigQuery. In OPEN mode the model can pass arbitrary SQL to query_thelook, including DDL/DML or multi-statement SQL billed to GOOGLE_CLOUD_PROJECT. Please add a local guard shared by dry_run/run_query/query_thelook that rejects anything except a single read-only SELECT before client.query() (and before the mock path too, so tests cover it).

[haiyuan-eng-google]

Fixed in b256f9c. Added warehouse.read_only_violation() (rejects non single-SELECT/WITH: DDL/DML, scripting keywords, multi-statement) and enforced it at the top of dry_run and run_query — before BigQuery AND before the mock — so query_thelook is covered too. Added test_read_only_guard_blocks_non_select.

[caohy1988]

This mentions --no-llm, but the argparse setup below only supports --beats. Either add the flag or remove this sentence and keep the documented deterministic command to CA_GOV_USE_BIGQUERY=0 python .../governance_demo.py --beats diff adversarial, which currently matches the implementation.

[haiyuan-eng-google]

Fixed in b256f9c. Removed the --no-llm sentence; the deterministic command is now CA_GOV_USE_BIGQUERY=0 ... --beats diff adversarial (matches the implementation). Also added a flexible beat to the driver.

[caohy1988]

Top-level demo polish suggestion: this is a strong leadership demo, but I would make the README follow the ADK sample shape a bit more closely before sharing broadly: add a small Mermaid graph for the three modes (STRICT -> golden workflow/refusal, FLEXIBLE -> golden first + validated promotion, OPEN -> agentic fallback), and add a short Related Guides/Related RFCs section pointing to the workflow/RFC google#92/google#93 material. That will make the control-plane story scannable for people who land directly on the sample, not only through the PR description.

[haiyuan-eng-google]

Thanks for the review — all six points addressed in b256f9c1.

README polish (this comment): added a Mermaid flowchart of the three governance modes (STRICT → refuse, FLEXIBLE → golden-first + validated promotion, OPEN → agentic fallback) and a Related section pointing to the engine (authored_workflow_spike / dynamic_supervisor_spike), RFC google#92/google#93, the sibling samples, and the CA verified-queries docs. The mode table now includes the FLEXIBLE beat.

Code review comments (replied inline):

1. FLEXIBLE is now a real, distinct live mode (flexible_registry + author_flexible_plan); OPEN reserved for the agentic fallback.
2. Sql carries question so it survives nl2sql → dry_run → run → freeze (promoted record keeps the original question).
3. The dry-run is now a real gate (branch on check.valid; invalid SQL → reject_invalid, nothing run or promoted).
4. New read_only_violation() guard rejects non single-SELECT before BigQuery and the mock, shared by dry_run/run_query/query_thelook.
5. Removed the nonexistent --no-llm doc; added a flexible driver beat.

Tests: 12 pass (added: flexible-gate-rejects-invalid, read-only guard, mode routing; flexible test asserts the promoted question is preserved).
Live re-validation (gemini-3.5-flash, global Vertex endpoint + real BigQuery): the FLEXIBLE miss generated SQL, passed the real dry-run gate, executed (Men $63.46 / Women $55.84), and promoted the query with its question intact.

[caohy1988]

Suggestion for LT demo repeatability: the default run includes the flexible beat, and that beat writes the promoted query into the file-backed ca_gov_store by default. After one rehearsal, rerunning the same default script will make beat 5 a governed hit instead of showing nl2sql -> dry_run -> promote, because the query is already in the pool. I would make the headless driver use a fresh temp CA_GOV_STORE by default (and print it for inspection), or add a --reset-store / --store flag and document the clean-start command. That keeps rehearsals deterministic while leaving adk web persistence intact.

[haiyuan-eng-google]

Fixed in 4e574f0. The headless driver now defaults to a fresh temp CA_GOV_STORE per run (printed as store: …), so the FLEXIBLE beat always shows nl2sql → dry_run → promote instead of becoming a governed hit on a re-run. Added --store (persist / share with adk web) and --reset-store (clear promoted, non-seed queries). adk web persistence is unchanged.

[caohy1988]

The talking track skips the FLEXIBLE beat in its numbered walkthrough, while the README and driver now run flexible before agentic. For the LT demo, I would align this narrative with the actual default order: add the average-sale-price flexible promotion beat here, then make the open-mode churn question beat 6. Otherwise the presenter notes miss the core middle-ground story this PR added.

[haiyuan-eng-google]

Fixed in 4e574f0. The numbered walkthrough now matches the README/driver order: beat 5 is the FLEXIBLE average-sale-price promotion (semantic-constrained nl2sql → real dry-run gate → run → promote), and the OPEN-mode churn question is beat 6.

[caohy1988]

Mock and real dry-run can diverge here: read_only_violation() already accepts WITH queries, and BigQuery dry-run would validate a legal CTE, but credential-less mode returns valid: false unless the qualified SQL starts with select. Since this demo is likely rehearsed with CA_GOV_USE_BIGQUERY=0, a valid nl2sql CTE can be rejected locally even though it would pass live. I would either treat the mock dry-run as valid after the read-only guard passes, or check select|with here, plus add a small test for warehouse.dry_run({'sql': 'WITH x AS (SELECT 1) SELECT * FROM x'})['valid'] is True.

[haiyuan-eng-google]

Fixed in 4e574f0. Mock and real dry-run no longer diverge: since read_only_violation() already confirms a single SELECT/WITH, the credential-less dry_run now returns valid: True once the guard passes (no second leading-select check). Added test_mock_dry_run_accepts_cte asserting dry_run({'sql': 'WITH x AS (SELECT 1) SELECT * FROM x'})['valid'] is True.

[caohy1988]

Small LT-demo doc polish: the driver now has the right behavior (fresh temp CA_GOV_STORE by default, plus --store / --reset-store), but this README section still only shows the old invocation. I would add one sentence here saying the headless driver uses a fresh store per run for repeatable rehearsals, and show the persistent/replay command, e.g. python .../governance_demo.py --store contributing/samples/workflows/authored_workflow_ca_governance_demo/ca_gov_store --reset-store. That makes it clear when beat 5 should re-promote versus when a presenter intentionally wants to share the promoted pool with adk web.

[haiyuan-eng-google]

Fixed in 904aff9. The Headless driver section now states the driver uses a fresh temp CA_GOV_STORE per run (printed as store: …) so beat 5 always re-promotes, and shows the persistent command for sharing the promoted pool with adk web:

python .../governance_demo.py --store contributing/samples/workflows/authored_workflow_ca_governance_demo/ca_gov_store --reset-store

[caohy1988]

With the new HITL flow, --reset-store should also clear pending_candidate.json, not just verified/. Otherwise a presenter can run with a durable --store, generate a candidate but not approve/reject it, then later run --reset-store expecting a clean rehearsal; the old pending candidate survives and a stray approve will promote stale SQL into the freshly reset pool. I would either set CA_GOV_STORE = store before this block and call golden.clear_pending() as part of reset, or remove os.path.join(store, "pending_candidate.json") directly here. The help/comments should also say the reset clears promoted + pending candidates in the HITL version.

[haiyuan-eng-google]

Fixed in the latest push. CA_GOV_STORE is now set before the reset block, and --reset-store clears both verified/ and the pending candidate via golden.clear_pending() — so a leftover candidate can't be approved into a freshly reset pool. The --reset-store help and README now say it clears promoted and pending. Verified: a pending candidate in a durable --store is gone after --reset-store.

[haiyuan-eng-google]

Update: the demo now genuinely exercises RFC google#93's model-authoring, not just the governance machinery. The plan is authored live by an LlmAgent(output_schema=WorkflowSpec) (_author_live, with retry), then validated against the registry and governed; a canned author_*_plan() is the deterministic fallback (and CA_GOV_LIVE_PLANNER=0 forces it). The banner shows 🧠 Model-authored (live) vs the fallback, honestly.

Verified live (gemini-3.5-flash, global Vertex + real BigQuery): the golden hit, the adversarial beat (the model's own nl2sql plan is rejected by STRICT), and the post-approval strict re-ask all author live; the flexible nested-gate plan falls back gracefully when the model emits an off-shape plan.

Implementation note for reviewers: ADK LlmAgent instructions treat {...} as session-state template vars, so the planner instructions are deliberately brace-free (the JSON schema comes from output_schema). Honest-scope: the plan shape is instruction-guided for on-camera reliability — free, un-prescribed authoring evidence remains in the sibling authored_workflow_spike / authored_workflow_demo samples. 18 tests pass.

[caohy1988]

Fresh feedback on the latest live-authoring update:

I think model-authored workflow is a strong fit for this demo. The demo is no longer only showing “workflow governance”; it now shows the more important RFC google#93 story: the model can author a typed WorkflowSpec, but the system still governs what that plan can compose, validates it, freezes it, and keeps promotion under human control.

The value is clearest in three beats:

• adversarial: the model-authored nl2sql -> run_adhoc plan is rejected under STRICT because the registry does not expose those capabilities;
• governed hit: even when the model authors the plan, execution is still from a frozen/auditable workflow over approved SQL;
• FLEXIBLE + HITL: the model can propose and validate a candidate, but cannot self-promote it into the governed pool.

That is the right enterprise story: “dynamic model-authored orchestration, bounded by structural policy,” not “please trust the model to follow a prompt.”

The remaining thing I would tighten before using this as the LT demo is the live-plan acceptance gate. Right now _author_live() accepts any registry-valid spec that contains the required ids. For the on-camera story, I would require the exact expected shape for golden/flexible/adversarial plans before labeling the turn as live model-authored; otherwise fall back. I left an inline comment here:

#9 (comment)

With that tightened, I think the demo shows the value of model-authored workflow well while staying honest about the current implementation: live authoring is real, but intentionally instruction-guided for reliability.

[caohy1988]

Preferred LT demo narrative:

I would make the story a governance-control story first, and a model-authoring story second. The key line is:

The model is allowed to author the workflow, but it is not allowed to choose its own powers.

Suggested flow:

1. Start with the enterprise ask.

   “Customers want Conversational Analytics, but some of them need a hard boundary: only answer from verified/golden queries unless policy explicitly allows more. A prompt like ‘only use verified queries’ is not governance. It is a request.”

2. Show the registry diff.

   “The boundary is structural. STRICT exposes only match_verified_query, run_frozen_query, summarize, and refuse. FLEXIBLE adds nl2sql, dry_run, run_adhoc, and reject_invalid. Neither registry exposes promotion. That means the model cannot invent a capability or write itself into the governed pool.”

3. Show the adversarial beat.

   “Now let the model author the wrong plan: nl2sql -> run_adhoc -> summarize. The plan may be model-authored, but validation rejects it under STRICT before any query runs. This is the headline: we are not trusting the model to obey a prompt; we are validating the workflow it authored against a capability registry.”

4. Show a governed hit.

   “For a verified question, the model authors a workflow, the workflow validates, freezes, and runs the approved SQL on BigQuery. The answer is dynamic in orchestration, but governed in execution: approved SQL, frozen spec hash, replayable artifact, zero model-drafted SQL on the governed path.”

5. Show STRICT refusal.

   “For an out-of-set question, STRICT refuses. That refusal is a feature. It proves the boundary fails closed: no verified match, no SQL run, no cost, no hallucinated answer.”

6. Show FLEXIBLE + HITL.

   “Some customers do not want a hard stop; they want constrained authoring. FLEXIBLE lets the model generate SQL under the allowed capability set, dry-run validates it, then it runs and parks the candidate. But the model still cannot promote it. A human must approve. After approval, the same question becomes a governed hit.”

7. Show OPEN.

   “OPEN is today’s normal agentic CA path: useful and powerful, but not frozen/auditable in the same way. The point is not that OPEN is bad; the point is that we can expose a dial: strict governed-only, flexible HITL-assisted authoring, or full agentic fallback.”

Close with:

“Model-authored workflow is valuable here because it separates who proposes the plan from who grants authority. The model authors; the registry limits; the validator enforces; the frozen record audits; the human approves promotion. That is the enterprise governance shape.”

One implementation caveat I would mention only if asked: the live authoring in this demo is intentionally instruction-guided for reliability. That is fine for LT, as long as the UI/text honestly distinguishes live model-authored plans from deterministic fallback and the acceptance gate verifies the expected shape.

[haiyuan-eng-google]

Tightened the live-plan acceptance gate as you asked (commit 51376c86).

_author_live() no longer accepts any registry-valid spec that merely contains the required node ids. The 🧠 Model-authored (live) label is now earned only when the authored plan matches the exact expected shape for that mode, via the predicates you suggested:

• _is_golden_shape, _is_flexible_shape, _is_adversarial_shape

Each compares a canonical structural signature — node order, ids, capability-per-id, input bindings, branch condition (on), route values, and the spec output binding — against the canned plan for that mode (so the predicates stay in sync with author_*_plan() automatically; single source of truth). Any registry-valid but off-shape plan now falls back to the deterministic canned plan and is honestly labeled a fallback — covering exactly the "right ids, wrong output binding / route / wiring" case you flagged.

Tests: 18 → 21. Added: shape predicates accept their own canned plan and reject cross-mode plans; an off-shape-but-registry-valid plan (right ids, different output binding) passes the validator but fails the shape gate; and with the live planner ON, an off-shape authored plan makes _author_live return None → caller uses the canned fallback.

README honest-scope updated to describe the exact-shape gate.

Live re-validated (gemini-3.5-flash, global Vertex + real BigQuery): the golden hit and the strict refusal both author live (the model reliably emits the exact shape, so the tightening doesn't cost the on-camera live label), and the adversarial plan is still rejected under STRICT. With this in, live authoring is real, honestly distinguished from fallback, and the acceptance gate verifies the expected shape — matching the narrative.

Also took your preferred LT narrative — the governance-control-first framing with "the model is allowed to author the workflow, but not to choose its own powers" — and worked it into the recording track. Thanks, this made the demo materially more honest.

[caohy1988]

Punchline: A human-compiled workflow hardcodes one policy path; a model-authored workflow lets the model adapt the plan to the question, while the registry prevents it from granting itself new authority.

Top 3 demo points:

1. Adaptive without losing control: the model can author the workflow for the user’s question, but it can only compose approved capabilities.
2. Governance is structural, not prompt-based: STRICT does not expose nl2sql, so even a model-authored SQL plan is rejected before anything runs.
3. Safe path from discovery to governance: FLEXIBLE lets the model generate and validate a candidate, but only human approve adds it to the governed pool.

[haiyuan-eng-google]

Love the punchline — adopted it (commit 96c8c46a, docs-only).

Folded the human-compiled-vs-model-authored framing and the three points into both NARRATIVE.md and README.md, told governance-first with your line front and center: the model is allowed to author the workflow, but not to choose its own powers. The "separates who proposes the plan from who grants authority" close is now the thesis of the talking track.

One small honesty edit I made on point 1 ("adaptive"): in this demo the plan shape is instruction-guided and now exact-shape-gated, so what the model adapts per question is the dial/mode, the match-vs-nl2sql branch it takes at runtime, and the SQL content — not free structural decomposition. I pointed to the sibling authored_workflow_spike / authored_workflow_demo samples for the unconstrained-authoring evidence, and noted that the governance guarantee (can't self-grant authority) holds regardless of authoring style — so the punchline lands without overclaiming on camera. Flag if you'd rather state it more strongly.