feature: Remove initContainer and Replace it with Kagenti Sidecar Containers#4
Merged
Conversation
Signed-off-by: Jarek Cwiklik <cwiklik@mac.lan>
18 tasks
51 tasks
4 tasks
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 7, 2026
Must-fix: - rossoctl#1+rossoctl#7: Plugins spawned credential watchers with initCtx (60s), so a slow client-registration orphaned the plugin after Start's deadline and OnRequest returned 503 for the pod's lifetime. Watchers now run on process-lifetime contexts created in Init via context.Background() + context.WithCancel; the cancel func is stored on the plugin and fired in Shutdown. Both JWTValidation and TokenExchange now implement pipeline.Shutdowner. Removed the orphaned context.WithCancel at main.go:89 that looked like it was meant to be that context. - rossoctl#2: testutil.go moved out of the plugins package into a dedicated sub-package authlib/plugins/plugintesting/ so it can't be imported by the production binary. Rewrote the helpers as stub plugin adapters that mimic jwt-validation / token-exchange OnRequest but don't touch plugin internals — listener tests inject an *auth.Auth via plugintesting.NewJWTValidation / NewTokenExchange, wire through plugintesting.BuildPipeline, and never see the real Configure path. All four listener tests updated. - rossoctl#11: Removed identity sub-validation that applyDefaults made unreachable; class-of-error compensated by logging a boot-time WARN when Configure's best-effort ReadCredentialFile fails (see also rossoctl#12). Suggestions: - rossoctl#3: pollCredentials no longer mutates p.cfg. Reads credential values into locals and feeds them through auth.UpdateIdentity. cfg is now immutable after Configure returns. Added a buildClientAuthFrom helper that takes explicit args so the goroutine doesn't read p.cfg either. - rossoctl#4: Rewrote the NoTokenPolicy doc comment to state the current behavior ("deny in all modes; operators who need allow or client-credentials must set it explicitly") instead of referring to the removed config.NoTokenPolicyForMode. - rossoctl#5: Deleted unused DefaultInboundPlugins/DefaultOutboundPlugins and the whole defaults.go file. Fixed the stale PipelineConfig docstring in config.go that said "defaults kick in when pipeline section is omitted" — they don't. - rossoctl#6: PluginEntry.UnmarshalYAML normalizes explicit `config: null` (and any other !!null-tagged scalar under `config:`) to a nil RawMessage, so Build's "plugin does not accept configuration" gate doesn't fire spuriously on the four bytes "null". - rossoctl#12: Deleted unreachable validate() branches: static-mode's audience emptiness check, spiffe's missing-path check, client-secret's missing-id / missing-secret checks. applyDefaults fills the matching field in every case; the branches were dead code that obscured what the plugin actually validated. Nits: - rossoctl#8: Renamed TestPluginEntry_IDDefaultsToName → TestPluginEntry_IDOmittedStaysEmpty. - rossoctl#9: TestBuild_ConfigForNonConfigurablePlugin now asserts the error text contains "does not accept configuration" — operator-facing contract. - rossoctl#10: Renamed TestConfigurable_ErrorAborts → TestConfigurable_ErrorPropagates and noted in the comment that Build-level abort coverage lives in plugins_test.go. - rossoctl#13: Documented on jwtValidationConfig.AudienceFile that empty- string is treated as "unset" and triggers the default. Operators who want no file backing must supply an explicit Audience. - rossoctl#14: Added TestAuthbridgeCombinedYAML_Loads — parses the in-repo authbridge-combined.yaml with env vars set, asserts both inbound and outbound pipelines Build cleanly. A future rename of any default constant that the YAML relies on breaks this test in CI rather than silently shipping a broken image. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 7, 2026
Should-fix items from the comprehensive PR evaluation. rossoctl#3 — WARN on NoTokenPolicy implicit default: The pre-migration default for no_token_policy was mode-dependent (envoy-sidecar=client-credentials, waypoint=allow, proxy-sidecar=deny). The migration hardcodes "deny" for all modes. Waypoint operators whose YAML doesn't explicitly set no_token_policy would silently start denying outbound requests that previously allowed. token-exchange.Configure now tracks whether NoTokenPolicy arrived explicitly, and emits a WARN at boot when the default was used AND default_policy is "exchange" (the configuration shape where the behavior change is visible). The WARN names the three valid values so operators can pin the behavior by adding one line to their YAML. rossoctl#4 — /stats stub visibility: The stat server returns empty counters because plugin-level stats aggregation is a follow-up. Without any signal, operators curling /stats see zeros and assume the service is broken. main.go now logs a WARN at the same point where it constructs the stub Stats object, explaining that /stats is a stub in this release but /config remains accurate. rossoctl#5 — Heartbeat during credential-file wait: WaitForCredentialFile emitted one WARN at Configure time when a file wasn't readable, then went silent. If the file never appeared (typo'd path, missing volume mount), operators chased 503s instead of seeing the wait in logs. It now emits a WARN every 60s (overridable via the package-scope heartbeatInterval, used from the test) while still waiting, carrying the path and total wait duration. Loud enough to notice during an operational incident, quiet enough to not drown out real signal. The test asserts the heartbeat branch of the select is reachable without panics. rossoctl#7 — Stale CLAUDE.md credential-wait description: The authbridge/CLAUDE.md gotcha bullet said "Authbridge waits up to 60s for [credential files]" — stale since the switch to indefinite background polling. Rewrote to match the new behavior and point at the heartbeat WARN as the diagnostic signal. rossoctl#8 — Reorder token-exchange Configure to avoid partial cfg mutation: Under the old ordering, p.cfg = c ran before clientAuth / router construction. If either failed, p.cfg was already set even though Configure returned an error. Since Configure errors take the process down today, this didn't manifest, but it was sloppy state management and a foot-gun for any future call path that Configure from non-startup contexts. Rewrote Configure to build clientAuth, router, and authCfg against the local `c` before any assignment to p.* fields. buildClientAuth and buildRouter methods removed; replaced with the pure buildClientAuthFrom and new buildRouterFrom functions that take explicit args. pollCredentials already used buildClientAuthFrom, so the migration is additive. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 7, 2026
rossoctl#1 atomic.Pointer for bgCancel: Both JWTValidation and TokenExchange now hold bgCancel in an atomic.Pointer[context.CancelFunc] rather than a plain field. Today's callers (pipeline.Start / pipeline.Stop) are serialized so there's no actual race, but a future caller invoking Shutdown from a different goroutine would have been latent-racing the Init assignment. Shutdown uses Swap(nil) so it's also idempotent. -race passes clean on pipeline, plugins, config packages. rossoctl#2 Widen NoTokenPolicy WARN: Previously only fired when default_policy was "exchange". That missed envoy-sidecar deployers relying on the old client-credentials default for routed hosts. WARN now fires whenever no_token_policy was defaulted, with the message listing all three pre-PR mode-specific defaults so operators can see which one their deployment used to get. rossoctl#3 audience_file WARN mentions defaulted origin: When audience_file was defaulted to /shared/client-id.txt (Kagenti convention) and the file isn't readable, the new WARN spells that out — so non-Kagenti deployers notice their config didn't put the plugin where it thinks it is. Explicit-path case keeps the shorter original message. rossoctl#4 Drop client_id from credential-loaded log: OAuth spec treats client_id as non-secret, but some operators treat it as sensitive. The signal (credentials loaded) doesn't need the identifier. rossoctl#5 TODO markers for /stats and Ready(): Grep-able TODO(follow-up) comments at the stub sites in main.go with sketch notes for the future work. These sit next to the existing behavior comments so anyone reading those surfaces finds the follow-up plan. rossoctl#6 Pipeline.Start failure unwind: On Init error at plugin N, Start now calls Shutdown on plugins [0..N-1] in reverse order before returning the error. Prevents leaking goroutines spawned by earlier plugins' Init when a downstream peer rejects its config at boot. Not a correctness bug for production (log.Fatalf then exit, kernel cleans up) but matters for any embedded or multi-tenant caller. New test TestPipelineStart_UnwindsSuccessfulInitsOnFailure asserts the reverse-order Shutdown on the subset that Init'd, and that plugins after the failing one are neither Init'd nor Shutdown'd. rossoctl#8 Document strictness asymmetry in CONVENTIONS.md: Plugin config is strict (DisallowUnknownFields); top-level runtime YAML is forgiving (unknown keys silently dropped). Note explains the forward-compat rationale and points at config.Validate's empty-pipeline guard as the mechanism that closes the obvious stale-schema gap. rossoctl#7 is procedural (kind smoke test) — not in code. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 7, 2026
Must-fix: - rossoctl#1+rossoctl#7: Plugins spawned credential watchers with initCtx (60s), so a slow client-registration orphaned the plugin after Start's deadline and OnRequest returned 503 for the pod's lifetime. Watchers now run on process-lifetime contexts created in Init via context.Background() + context.WithCancel; the cancel func is stored on the plugin and fired in Shutdown. Both JWTValidation and TokenExchange now implement pipeline.Shutdowner. Removed the orphaned context.WithCancel at main.go:89 that looked like it was meant to be that context. - rossoctl#2: testutil.go moved out of the plugins package into a dedicated sub-package authlib/plugins/plugintesting/ so it can't be imported by the production binary. Rewrote the helpers as stub plugin adapters that mimic jwt-validation / token-exchange OnRequest but don't touch plugin internals — listener tests inject an *auth.Auth via plugintesting.NewJWTValidation / NewTokenExchange, wire through plugintesting.BuildPipeline, and never see the real Configure path. All four listener tests updated. - rossoctl#11: Removed identity sub-validation that applyDefaults made unreachable; class-of-error compensated by logging a boot-time WARN when Configure's best-effort ReadCredentialFile fails (see also rossoctl#12). Suggestions: - rossoctl#3: pollCredentials no longer mutates p.cfg. Reads credential values into locals and feeds them through auth.UpdateIdentity. cfg is now immutable after Configure returns. Added a buildClientAuthFrom helper that takes explicit args so the goroutine doesn't read p.cfg either. - rossoctl#4: Rewrote the NoTokenPolicy doc comment to state the current behavior ("deny in all modes; operators who need allow or client-credentials must set it explicitly") instead of referring to the removed config.NoTokenPolicyForMode. - rossoctl#5: Deleted unused DefaultInboundPlugins/DefaultOutboundPlugins and the whole defaults.go file. Fixed the stale PipelineConfig docstring in config.go that said "defaults kick in when pipeline section is omitted" — they don't. - rossoctl#6: PluginEntry.UnmarshalYAML normalizes explicit `config: null` (and any other !!null-tagged scalar under `config:`) to a nil RawMessage, so Build's "plugin does not accept configuration" gate doesn't fire spuriously on the four bytes "null". - rossoctl#12: Deleted unreachable validate() branches: static-mode's audience emptiness check, spiffe's missing-path check, client-secret's missing-id / missing-secret checks. applyDefaults fills the matching field in every case; the branches were dead code that obscured what the plugin actually validated. Nits: - rossoctl#8: Renamed TestPluginEntry_IDDefaultsToName → TestPluginEntry_IDOmittedStaysEmpty. - rossoctl#9: TestBuild_ConfigForNonConfigurablePlugin now asserts the error text contains "does not accept configuration" — operator-facing contract. - rossoctl#10: Renamed TestConfigurable_ErrorAborts → TestConfigurable_ErrorPropagates and noted in the comment that Build-level abort coverage lives in plugins_test.go. - rossoctl#13: Documented on jwtValidationConfig.AudienceFile that empty- string is treated as "unset" and triggers the default. Operators who want no file backing must supply an explicit Audience. - rossoctl#14: Added TestAuthbridgeCombinedYAML_Loads — parses the in-repo authbridge-combined.yaml with env vars set, asserts both inbound and outbound pipelines Build cleanly. A future rename of any default constant that the YAML relies on breaks this test in CI rather than silently shipping a broken image. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 7, 2026
Should-fix items from the comprehensive PR evaluation. rossoctl#3 — WARN on NoTokenPolicy implicit default: The pre-migration default for no_token_policy was mode-dependent (envoy-sidecar=client-credentials, waypoint=allow, proxy-sidecar=deny). The migration hardcodes "deny" for all modes. Waypoint operators whose YAML doesn't explicitly set no_token_policy would silently start denying outbound requests that previously allowed. token-exchange.Configure now tracks whether NoTokenPolicy arrived explicitly, and emits a WARN at boot when the default was used AND default_policy is "exchange" (the configuration shape where the behavior change is visible). The WARN names the three valid values so operators can pin the behavior by adding one line to their YAML. rossoctl#4 — /stats stub visibility: The stat server returns empty counters because plugin-level stats aggregation is a follow-up. Without any signal, operators curling /stats see zeros and assume the service is broken. main.go now logs a WARN at the same point where it constructs the stub Stats object, explaining that /stats is a stub in this release but /config remains accurate. rossoctl#5 — Heartbeat during credential-file wait: WaitForCredentialFile emitted one WARN at Configure time when a file wasn't readable, then went silent. If the file never appeared (typo'd path, missing volume mount), operators chased 503s instead of seeing the wait in logs. It now emits a WARN every 60s (overridable via the package-scope heartbeatInterval, used from the test) while still waiting, carrying the path and total wait duration. Loud enough to notice during an operational incident, quiet enough to not drown out real signal. The test asserts the heartbeat branch of the select is reachable without panics. rossoctl#7 — Stale CLAUDE.md credential-wait description: The authbridge/CLAUDE.md gotcha bullet said "Authbridge waits up to 60s for [credential files]" — stale since the switch to indefinite background polling. Rewrote to match the new behavior and point at the heartbeat WARN as the diagnostic signal. rossoctl#8 — Reorder token-exchange Configure to avoid partial cfg mutation: Under the old ordering, p.cfg = c ran before clientAuth / router construction. If either failed, p.cfg was already set even though Configure returned an error. Since Configure errors take the process down today, this didn't manifest, but it was sloppy state management and a foot-gun for any future call path that Configure from non-startup contexts. Rewrote Configure to build clientAuth, router, and authCfg against the local `c` before any assignment to p.* fields. buildClientAuth and buildRouter methods removed; replaced with the pure buildClientAuthFrom and new buildRouterFrom functions that take explicit args. pollCredentials already used buildClientAuthFrom, so the migration is additive. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 7, 2026
rossoctl#1 atomic.Pointer for bgCancel: Both JWTValidation and TokenExchange now hold bgCancel in an atomic.Pointer[context.CancelFunc] rather than a plain field. Today's callers (pipeline.Start / pipeline.Stop) are serialized so there's no actual race, but a future caller invoking Shutdown from a different goroutine would have been latent-racing the Init assignment. Shutdown uses Swap(nil) so it's also idempotent. -race passes clean on pipeline, plugins, config packages. rossoctl#2 Widen NoTokenPolicy WARN: Previously only fired when default_policy was "exchange". That missed envoy-sidecar deployers relying on the old client-credentials default for routed hosts. WARN now fires whenever no_token_policy was defaulted, with the message listing all three pre-PR mode-specific defaults so operators can see which one their deployment used to get. rossoctl#3 audience_file WARN mentions defaulted origin: When audience_file was defaulted to /shared/client-id.txt (Kagenti convention) and the file isn't readable, the new WARN spells that out — so non-Kagenti deployers notice their config didn't put the plugin where it thinks it is. Explicit-path case keeps the shorter original message. rossoctl#4 Drop client_id from credential-loaded log: OAuth spec treats client_id as non-secret, but some operators treat it as sensitive. The signal (credentials loaded) doesn't need the identifier. rossoctl#5 TODO markers for /stats and Ready(): Grep-able TODO(follow-up) comments at the stub sites in main.go with sketch notes for the future work. These sit next to the existing behavior comments so anyone reading those surfaces finds the follow-up plan. rossoctl#6 Pipeline.Start failure unwind: On Init error at plugin N, Start now calls Shutdown on plugins [0..N-1] in reverse order before returning the error. Prevents leaking goroutines spawned by earlier plugins' Init when a downstream peer rejects its config at boot. Not a correctness bug for production (log.Fatalf then exit, kernel cleans up) but matters for any embedded or multi-tenant caller. New test TestPipelineStart_UnwindsSuccessfulInitsOnFailure asserts the reverse-order Shutdown on the subset that Init'd, and that plugins after the failing one are neither Init'd nor Shutdown'd. rossoctl#8 Document strictness asymmetry in CONVENTIONS.md: Plugin config is strict (DisallowUnknownFields); top-level runtime YAML is forgiving (unknown keys silently dropped). Note explains the forward-compat rationale and points at config.Validate's empty-pipeline guard as the mechanism that closes the obvious stale-schema gap. rossoctl#7 is procedural (kind smoke test) — not in code. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
This was referenced May 11, 2026
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 12, 2026
…mics Addresses five review comments from PR rossoctl#401, four code changes plus one comment-only tweak: rossoctl#1 — Framework-stamped rejecting plugin (robustness). OutcomeFromContext previously inferred the denying plugin from pctx.Extensions.Invocations, making correctness dependent on plugin authors pairing Action{Type: Reject} with a pctx.Record call. A plugin that returned Reject without recording was misclassified as OutcomeAllow or OutcomeError. Pipeline.Run / RunResponse now stamp pctx.rejectingPlugin on every enforce-policy Reject. OutcomeFromContext reads that first; the Invocation walk stays as defense-in-depth for bespoke dispatchers. Shadow rejections (on_error: observe) do not stamp the field — matches "the pipeline effectively allowed" semantics already used by abctl. Exposed via pctx.RejectingPlugin() getter for listeners that build Outcome explicitly. rossoctl#2 — context.WithoutCancel in dispatchFinish (ergonomics). OnFinish previously ran under context.Background()-derived ctx, discarding any values (slog fields, request ID, tracing span) the caller carried. Now uses context.WithoutCancel(parent) as the base: preserves values while still detaching cancellation so client disconnect doesn't abort cleanup I/O. The caller-supplied ctx is wired through RunFinish; its parameter is no longer vestigial. rossoctl#3 — Drop ext_authz StatusCode=200 sentinel (cleanup). OutcomeFromContext classifies pctx.StatusCode == 0 as OutcomeError, which is correct for HTTP listeners where 0 means "no response written." ext_authz has no HTTP status concept, so it was setting StatusCode=200 as a sentinel to coerce the helper into returning OutcomeAllow. With rossoctl#1 landed, ext_authz now calls a local authzOutcome(pctx) helper that reads pctx.RejectingPlugin() directly and defaults to OutcomeAllow — no sentinel, intent is explicit. rossoctl#4 — RunFinish double-call guard (defense-in-depth). A pctx.finished flag, set at RunFinish entry, causes a second call to log a WARN and early-return rather than double-release every Finisher's per-request state. All current listeners call RunFinish exactly once; this is purely protection against a refactor accident (two defers registered) or dispatch bug. rossoctl#5 — Reword ext_authz LIFO comment (readability). Moved the LIFO note to the outbound defer's block; the inbound defer now just says "fires whether the pipeline allows, denies, or Check returns early." The prior wording made the inbound defer's comment easy to misread as "inbound runs first." New tests: TestRunFinish_DoubleCallGuard second call is no-op TestRunFinish_CtxValuePropagation values carry through detach TestRun_StampsRejectingPlugin allow/enforce-deny/shadow-deny TestOutcomeFromContext_PrefersRejectingPlugin plugins that return Reject without recording are still Deny Full authlib + cmd/authbridge test suites pass under -race. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
7 tasks
huang195
added a commit
to huang195/kagenti-extensions
that referenced
this pull request
May 29, 2026
Three suggestions worth addressing:
1. Eager Catalog() at boot. New plugins.WarmCatalog() helper is
called in each binary's main() right before sessionapi.New, so any
factory that violates the constructor contract (panics, allocates
goroutines/connections/file handles) fails fast at startup rather
than silently caching a faulty throwaway instance on the first
/v1/plugins request. Cheap insurance — the result is already
memoized so this just moves "first call" out of the request hot
path.
2. Stale-catalog hint on the editor's "Unknown plugin" validation
error. abctl caches /v1/plugins for the session; a freshly-
installed plugin in-cluster won't appear until refresh. The
message now hints at the staleness path:
Unknown plugin "foo" (not in cached /v1/plugins; catalog may
be stale, press P then r to refresh)
The prompt is already non-blocking ("apply anyway? (y/N)"), so
operators can still proceed; this just saves one round of
confusion.
3. Document the auth-posture decision on /v1/plugins. The handler
inherits the no-auth posture of the rest of /v1/*; the catalog
reveals plugin metadata (names, deps, descriptions) but never
user content or secrets, so it's fine. Doc-only addition so the
next maintainer doesn't re-derive the conclusion.
Reviewer's rossoctl#4 was a compliment on the inboundSessionID comment
block — no action needed.
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
oblinder
added a commit
to s-and-p-team/kagenti-extensions
that referenced
this pull request
Jun 29, 2026
- Add RAG Ingest → ChromaDB arrowhead in PRD.md high-level diagram - Replace arch diagram in ARCHITECTURE-SUMMARY with PRD version (Policy Store Pod, Policy Compute Engn box, RAG Ingest ──► ChromaDB) - Update component table: 7 → 8 entries; Policy Management Service → Policy Store (rossoctl#3); add Policy Computation Engine (rossoctl#4); update library module paths and descriptions throughout Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Oleg Blinder <olegb@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR removes code which injects initContainer from MCPServer Custom Resource in favor of injecting Kagenti sidecars.
The two Kagenti sidecars are: