AE control auth + CodeRabbit #53 followup - #68
Merged
Merged
Conversation
Signed-off-by: entlein <einentlein@gmail.com>
New env-gated background loop that runs the same PxL shape AE's anomaly-gated path uses, but with an empty Target (no ns/pod predicate) and over a configurable rolling window. Writes via the existing sink so the byte-shape of forensic_db rows is comparable between the PASSTHROUGH=1 phase (EVERYTHING) and the PASSTHROUGH=0 phase (AE-FILTER). One-shot A/B that yields the per-table capture fraction of the adaptive write path. - internal/passthrough/passthrough.go — Loop + Config; defaults to 30s window / 30s refresh / clickhouse.PixieTables() table list. - internal/passthrough/passthrough_test.go — 6 tests; the load-bearing one is TestLoop_EmitsEmptyTargetPxL (asserts neither df.namespace nor df.pod predicates appear in the emitted PxL). - cmd/main.go: ADAPTIVE_PASSTHROUGH + _WINDOW_SEC + _REFRESH_SEC env knobs. Adapter is constructed unconditionally when passthrough is on (joins the existing PushPixie / streaming construction path so the same pxapi grpc stream is reused). Loop is registered with the shutdown WaitGroup so SIGTERM waits for the in-flight tick. - cmd/BUILD.bazel: drop @px// load (other AE BUILD.bazel files use //bazel — sticking out as the only one with @px is a leftover from a prior gazelle run; align). Add passthrough dep. Signed-off-by: entlein <einentlein@gmail.com>
Stand-alone workflow that builds entlein/dx (private Active-Diagnosis Framework) into ghcr.io/k8sstormcenter/dx-daemon. Separates the dx image publish from the bazel-based vizier_release pipeline; the dx repo ships its own Dockerfile.dxd (Go cross-compile + distroless final stage) so it doesn't need to live as a submodule inside src/vizier/services/dx. Triggers: - tag push 'release/dx/v*' on this repo cuts a release build, image tag derived from the tag suffix (release/dx/v0.1.0 -> image tag 0.1.0). - workflow_dispatch lets us build any dx ref on demand with a custom tag (default: short sha of the resolved dx commit). Pulls dx via DX_ENTLEIN_PAT (already configured on the repo). Multi-arch build (linux/amd64, linux/arm64); Dockerfile.dxd cross-compiles in the native BUILDPLATFORM stage and the final stage is COPY-only, so target emulation isn't required. Signed-off-by: entlein <einentlein@gmail.com>
The dx image build pipeline lives in entlein/dx itself (PR #53, branch feat/bazel-release): bazel-based with @px external pin to pixie's ae-prod tip, pushes to docker.io/entlein/dx-daemon on a release/dx/v* tag in the dx repo. The pixie-side buildx workflow this reverts duplicated that intent in the wrong repo + the wrong build system (docker buildx instead of bazel + pl_go_image macros) + the wrong registry (ghcr.io/k8sstormcenter instead of docker.io/entlein). Signed-off-by: entlein <einentlein@gmail.com>
…affordances Fixes the silent-halt bug: the trigger gated on a RAW event_time high-water-mark, so a single anomaly in a larger unit (ms/ns) drove the watermark past all real seconds rows and AE stopped processing forever (data still on Pixie). Normalize event_time to canonical nanoseconds in the poll SELECT filter+order and in the in-memory/persisted cursor, boundary-dedup, and maxSeen (normalizeEventTimeNanos + chNormEventTimeNanos). Validated at the data layer: vs a poisoned watermark the raw filter returns 0 rows, the normalized filter recovers all 60. Also adds ADAPTIVE_PUSH_REFRESH_SEC (negative = single-shot pull) for the reproducible load-test harness, an in-package trigger unit test, and an e2e hermetic load test (mock PixieQuerier, exact rows+bytes). Signed-off-by: entlein <einentlein@gmail.com>
…ness
Consolidate the adaptive_export load-test harness under src/e2e_test/ (matching
vzconn_loadtest / px_cluster conventions). Control-plane experiments (E1-E4, E6,
E8 sustained) are proven exactly-reproducible on a live rig; the data-plane
experiments (E5, E8 data-mode) are authored and pending live validation on a
vizier-registered rig (status documented in README + FINDINGS_AND_BACKLOG).
- harness/: shell + python (inject, exp_control, exp_e8, stats, ...) + lib helpers
- fixtures/EXPERIMENTS.md: curated kubescape_logs data-set catalog + expected outputs
- k8s/: isolated sinks + per-rep generator pod (no probes)
- tools/loadgen/: cleanloadgen + httpsink (docker-built test tool; .bazelignore'd
pending a bazel target — lib/pq is already vendored in the module)
- FINDINGS_AND_BACKLOG.md: F8 watermark-poison bug + the fix + AE backlog
The AE Go unit/e2e tests live with the service (internal/{trigger,e2e}).
Signed-off-by: entlein <einentlein@gmail.com>
…C14) + diagrams Signed-off-by: entlein <einentlein@gmail.com>
…iagram; gen sustained-DNS mode C15 = AE must keep re-pulling+writing an active pod until t_end or DX stop (the contract DX steers on; last week's 'wrote then stopped' is its violation). Add DX-steering sequence diagram. Generator gains SUSTAIN_SEC (distinct-DNS trickle, a Pixie-traced protocol) + configurable SETTLE_PRE_MS warm-up for fresh-pod capture; harness wires GEN_SUSTAIN_SEC/GEN_SETTLE_MS. Signed-off-by: entlein <einentlein@gmail.com>
…lisation The 700821d trigger unit-normalisation wrapped event_time in a multiIf(...) inside both the WHERE filter and the ORDER BY. Three existing tests in watermark_test.go + one in clickhouse_test.go pinned the raw 'event_time >= N' substring and broke at HEAD. Update each test's expected substring to match the new normalized form (') >= <ns-scaled N>' — the closing paren of multiIf, then the value in canonical nanoseconds). Per-test ns-scaling: watermark_test.go:94 1744000000000000000 already ns -> unchanged watermark_test.go:125 InitialWatermark=42 < 1e10 sec -> * 1e9 watermark_test.go:156 InitialWatermark=7 < 1e10 sec -> * 1e9 watermark_test.go:297 event_time='5000' < 1e10 sec -> * 1e9 clickhouse_test.go:82 ref.T=1744..303e9 ns already ns -> unchanged go test ./src/vizier/services/adaptive_export/... all green. Signed-off-by: entlein <einentlein@gmail.com>
…future-stamp watermark poison) Signed-off-by: entlein <einentlein@gmail.com>
Records one forensic_db.ae_reconcile row per data-plane pull (read_count vs wrote_count, window, ns/pod) across ALL three write paths — controller fan-out (filter), passthrough firehose, and streaming scanner — so a reconcile run localizes loss to query (R5: read<PEM) vs sink (R6: wrote<read) and quantifies re-pull dup (C8). Counts alone (write >= read) were proven insufficient. - new internal/reconcile leaf package (Row, Recorder, Nop) — no import cycle - sink.Record: CH-backed recorder (INSERT INTO forensic_db.ae_reconcile) - ae_reconcile table: schema.sql + KnownTables + OperatorOwnedTables (synced); not a pixie table (absent from PixieTables, so VerifyPixieSchema ignores it) - wired: passthrough.tick, controller.pushPixieRows (deferred, all return paths), streaming.scanner.Run; gated by ADAPTIVE_RECONCILE=true, else Nop - unit test proves read/wrote capture incl. the sink-drop read>wrote shape - fixed apply_test trailing-tables guard for the new operator table - harness: exp_row_reconcile.sh (row-level PEM<->CH), ae_vs_all.sh, exp_datavolume_extreme.sh Signed-off-by: entlein <einentlein@gmail.com>
px -o json empty result previously printed one blank line → counted as a phantom LOSS=1. Guard: empty set → 0-byte keys file; drop all-empty-field keys. Confirmed against the controlled log4j run (backend http 14/14 exact, conn 66>=12, no loss). Signed-off-by: entlein <einentlein@gmail.com>
…log4shell Reliable BY CONSTRUCTION against bob#140 (stateful/unreliable exploit on re-fire): fresh-JVM backend (delete pod) + attacker-before-backend + the WORKING resolvable FQDN attacker.<ns>.svc.cluster.local:1389 (NOT the bare attacker-ns.svc which NXDOMAINs and gets dropped), then VERIFY the actual backend->:1389 LDAP egress in forensic_db.conn_stats and RETRY until confirmed (the validity gate). Never assumes the exploit fired. Node-side. Signed-off-by: entlein <einentlein@gmail.com>
…tion) Reword from offensive 'exploit' to detection-signal-generation language: this validates the kubescape->DX->AE detection chain. No logic change. Signed-off-by: entlein <einentlein@gmail.com>
… http2 - pxl.CompilePassthrough/Render: precompile per-table PxL once (fixed window => constant relative start_time), only the two time_ bounds are stamped per tick. Rendered output is byte-identical to QueryFor with an empty Target (TestCompilePassthrough_MatchesQueryFor), so this is a structural change, not a capture change. upid->pod/ns stays in PxL. - passthrough: tickConcurrent fans every table out at once (was a serial loop); shared pull() helper. Sink/recorder are pool/HTTP-backed and already called concurrently elsewhere. - drop http2_messages.beta from the firehose set (not materialised on every cluster => ""Table not found"" every tick); shared PixieTables/DDL lists untouched. - toggle ADAPTIVE_PASSTHROUGH_COMPILED (default on; =false reverts to the legacy serial QueryFor path). Signed-off-by: entlein <einentlein@gmail.com>
…e.go Fixes "missing strict dependencies: import of .../internal/reconcile" that broke the AE image build for passthrough, sink, streaming, controller, cmd (pre-existing since the ADAPTIVE_RECONCILE commit added the package + imports without bazel deps; never CI-built). Also wires the new pxl/compile.go srcs + passthrough/pxl test srcs (compiled_test.go, reconcile_test.go, compile_test.go). - new internal/reconcile/BUILD.bazel (go_library, stdlib-only) - +//internal/reconcile dep: passthrough, sink, streaming, controller, cmd - pxl go_library +compile.go; pxl_test +compile_test.go; passthrough_test +compiled_test.go,reconcile_test.go (+reconcile dep) Signed-off-by: entlein <einentlein@gmail.com>
F1 RCA: Pixie caps every px.display at max_output_rows_per_table (default 10000, query_flags.go) — the planner add_limit_to_batch_result_sink_rule silently truncates wide firehose windows / busy pods at the READ (write path is clean: ae_reconcile shows read==wrote). Fix uses Pixie own native knob — prepend `#px:set max_output_rows_per_table=1000000` to every generated PxL (QueryFor + CompilePassthrough) so all pull paths are uncapped. Validated on rig: a 14208-row window returned 10000 (capped) vs 14298 (with flag). No pagination loop, no extra round-trips. See memory project-ae-passthrough-10k-cap. Signed-off-by: entlein <einentlein@gmail.com>
Consolidates the recurring content_type silent-drop incident class into one default-suite test gate (6 tests, ~15ms): I1 TestContract_ContentTypeIsInt64InSchema I2 TestContract_FastEncodeContentTypeAsInt I3 TestContract_SilentDropDetected I3.b TestContract_SilentDropNotTriggeredOnSuccess I3.c TestContract_SilentDropToleratesMissingSummaryHeader I4 TestContract_HTTPEventsRoundTrip Top-of-file docstring chronicles the incident timeline so future operators can grep their way to the contract. Signed-off-by: entlein <einentlein@gmail.com>
Measures the AdaptiveExport value prop: datavolume REDUCTION of DX-steered AE (rev-3 streaming, AE writes only DX-steered activeSet pods over the control surface) vs saving ALL data (passthrough firehose). Two arms, same fixed load, forensic_db active-part deltas (rows+bytes) per table; reduction = 1 - DX/ALL. Uses the canonical resolvable JNDI FQDN (attacker.attacker-ns.svc.cluster.local) so the chain fires + DX can classify (a malformed host → NXDOMAIN → no steer). Successor to ae_vs_all.sh, whose AE arm used the rev-2 controller gate + stale JNDI. Signed-off-by: entlein <einentlein@gmail.com>
Measures all AE non-functional requirements under steady load on the rig: throughput (rows+bytes/sec), capture completeness (AE read vs broker count = F1 cap proof), write fidelity (read==wrote + write-error count), end-to-end freshness latency (now - max(time_) in CH), resource footprint (AE pod cpu/mem idle vs loaded), per-cycle cadence. Emits a structured report; companion to exp_dx_steering_reduction.sh. Real-data only. Signed-off-by: entlein <einentlein@gmail.com>
…ring + live-pod guard) Run-1 reported false 100% reduction because stale adaptive_attribution windows rehydrated DEAD pods (deleted loaders) into the activeSet → AE streamed dead pods → 0 rows. Clear adaptive_attribution before the DX arm so the activeSet only gets freshly-steered LIVE pods; add a guard that prints the steered pods + marshalsec fire count + live log4j-poc pods so a dead-arm result is caught, not reported as a reduction. Signed-off-by: entlein <einentlein@gmail.com>
lag query used now()-DateTime64 (type error -> na); use dateDiff(second,...). Capture-completeness vs broker was window-misaligned (623% artifact) -> drop it; report tot_read vs tot_wrote (read==wrote) + errs instead. The 10k-cap/completeness proof is the dedicated F1 test (max_read>10000 vs broker for the SAME window). Signed-off-by: entlein <einentlein@gmail.com>
…ary) Run-2 byte-delta reduction came out negative because system.parts byte delta is compaction-noisy (merges land mid-window). Report rows reduction as primary (actual captured-row count, noise-free); keep bytes as secondary context. Signed-off-by: entlein <einentlein@gmail.com>
Eviction-RCA finding (PR #63 NFR campaign): AE had NO memory limit (only cpu 300m) and was CPU-pinned at 300m under concurrent passthrough. AE measured tiny (16-38Mi steady), but the raised 1M-row passthrough cap can spike, so cap at 1Gi so AE can never memory-pressure a node; raise cpu limit 300m->1 core (was throttling). NOTE node evictions were NOT AE/OOM — node-01 went NotReady (network/heartbeat); the memory consumer is PEM (1365Mi, OOMs at the 2Gi default). Signed-off-by: entlein <einentlein@gmail.com>
Root cause of the recurring "AE unauthenticated / writes 0 / crashloop" reverts: kustomization.yaml bundled adaptive_export_secrets.yaml (placeholder pixie-api-key) with the role+deployment, so EVERY infra re-apply (make log4j) clobbered the real key that ae-auth had written. Separation of concerns: remove the secret from the kustomization — infra (role+deployment) stays re-appliable; the secret holds real creds and is owned solely by `make ae-auth`, created once, never touched by infra re-applies. Secret manifest kept as a hand-applied seed-only template (documented). Signed-off-by: entlein <einentlein@gmail.com>
The DX-steered arm was failing because the harness only fired stage-1 (JNDI/LDAP). That generates ldap-egress but NO kubescape R0001 → backend never flagged → DX no case → indeterminate → AE steers wrong/no pods. R0001 comes from stage-2 (post- exploitation exec). fire() now does stage-1 (JNDI) + stage-2 (whoami/shadow/token/ getent in the backend) → kubescape R0001+R0006 → DX rules backend MALIGNANT → backend enters AE activeSet → reduction is measurable. Verified live: DX evidence unexpected-spawn+sensitive-file-read → verdict ruled_in generic=MALIGNANT. Signed-off-by: entlein <einentlein@gmail.com>
…dd DX-steering diagnostics
Standing terminology rule: allowlist/blocklist, never whitelist/blacklist.
Pure rename (no behavior change) of the rev-3 streaming filter:
FilterModeWhitelist → FilterModeAllowlist
MaxWhitelistSize → MaxAllowlistSize
ADAPTIVE_STREAM_MAX_WHITELIST → ADAPTIVE_STREAM_MAX_ALLOWLIST (env)
mode=whitelist log string → mode=allowlist
plus all comments/identifiers/tests in streaming, activeset, cmd/main.
DX-steering diagnostics (the reason DX-arm-writes-0 has been hard to RCA —
we could not tell "empty ActiveSet" from "broker returned 0 rows"):
- scanner: log the empty-allowlist short-circuit (was silent) so an
empty ActiveSet is visible in logs, distinct from "query completed rows=0".
- FilterUpdater: emitted-filter log Debug→Info so the steered pod count
per ActiveSet change is visible without debug logging.
NOTE: ADAPTIVE_STREAM_MAX_WHITELIST env renamed → tooling that sets the old
name must switch to ADAPTIVE_STREAM_MAX_ALLOWLIST.
Signed-off-by: entlein <einentlein@gmail.com>
The Pixie dx_evidence_graph UI reads dx_attack_graph via px.DataFrame clickhouse_dsn, whose query template hardcodes event_time + hostname and ORDER BY event_time. A table without those columns fails 'Unknown identifier event_time'; a table created by hand (local, not via the operator) isn't globally registered. Fix: make AE own it like the other forensic tables. - schema.sql: dx_attack_graph DDL with event_time(UInt64 nanos) + hostname, edge columns, fromUnixTimestamp64Nano partition/TTL (nanos-correct). - KnownTables + OperatorOwnedTables: register it so Apply creates it at boot. - apply_test: assert last-applied DDL == last OperatorOwnedTables entry (robust to appended operator tables) instead of hardcoding trigger_watermark. go test ./.../clickhouse green. Signed-off-by: entlein <einentlein@gmail.com>
Pixie's clickhouse_dsn type mapper reads UInt8 as BOOLEAN and does not handle UInt16/UInt32/Float32 -> px fails with 'Column[N] given incorrect type' rendering the dx_evidence_graph. weight/max_severity/num_findings -> Int64, confidence -> Float64 (map cleanly to INT64/FLOAT64). Verified live: px run returns all 6 edges with every column. event_time stays UInt64 (matches kubescape_logs, which px reads). Signed-off-by: entlein <einentlein@gmail.com>
…canner buildPxL The DX/streaming arm silently capped each per-table pull at Pixie's default 10000-row limit while the passthrough/ALL arm (pxl.CompilePassthrough / QueryFor) already raises it to 1,000,000 via the broker's #px:set query flag. Validated live on 6a33dac0: a single streaming http_events pull returned exactly rows=10000 (the cap). Left unfixed this UNDER-counts the DX arm and OVERSTATES the DX-vs-ALL volume reduction. Prepend the same #px:set directive to the streaming scanner's PxL so both arms are uncapped and comparable. Signed-off-by: entlein <einentlein@gmail.com>
# Conflicts: # skaffold/skaffold_cloud.yaml # src/e2e_test/adaptive_export_loadtest/README.md
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR expands adaptive export’s ClickHouse schema, control-surface security, trigger polling, persistence handling, streaming behavior, and live Go load-test suite. It also mounts TLS certificates, adds a direct TLS client option, updates runner labels, and excludes an unconverted load generator from Bazel. ChangesAdaptive export runtime and validation
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
added 4 commits
July 16, 2026 12:48
…fixtures + KPIs) Collapse the 12-script shell harness (+ stats.py) into a table-driven Go test suite under suite/: fixtures (control-plane experiments), KPI asserts via testify/require (reproducibility/reconcile/reduction), and one runner driving a deployed AE image on a rig (live-gated by AELOAD_LIVE). Removes all CVE names and adversarial verbs from the harness; kubescape/k8s wire tokens stay literal.
Volume reduction is a measured outcome to report, not an invariant — asserting 'reduction >= minPct' gates on a made-up threshold. Removed RequireReductionAtLeast; the reduction test now measures + logs the firehose->steered delta without a pass/fail gate.
…test) kubescape_logs.event_time is unix NANOSECONDS (Vector kubescape_enrich emits ns). The DDL read it as seconds via toDateTime() -> ns overflow -> wrong partitions + rows born already-TTL-expired (the F1/F8 unit bug). Fix, matching soc clickhouse-lab: PARTITION BY toYYYYMM(fromUnixTimestamp64Nano(event_time)) TTL toDateTime(fromUnixTimestamp64Nano(event_time)) + INTERVAL 30 DAY Also raise the protocol tables' derived event_time DateTime64(3) (ms) -> DateTime64(9) (ns), matching time_. The loadtest suite now injects UnixNano and documents C1 as ns.
Author
|
@ConstanzeTU build-agent — could you build a TTL AE image (and PEM if handy) from this PR's head so we can test the nanosecond pipeline?
Thanks! |
Author
|
build-agent: please build a TTL AE image from this branch's head so we can test.
(rigs currently run aeprod23 = old ms/seconds schema; this is the blocker for the nanos test.) |
|
🚧 On it — building the TTL AE image from
I'll reply here with the pullable ref the moment it publishes. PEM image can follow if you still want it — say the word. (Apologies for the delay — my watcher missed your 16:59 request.) |
|
✅ TTL AE image ready (local build → ttl.sh, no release needed):
Pull/deploy and run the AE control-plane loadtest suite against it. Ping if you want a PEM image too, or a longer TTL / a durable |
added 5 commits
July 16, 2026 22:39
…+ loadtest) + AELOAD_REPS knob evidence.sh captures, on a live rig: pipeline components Running (kubescape→vector→ node-agent→ClickHouse→AE), every forensic_db table created, event_time NANOSECONDS end-to-end (kubescape_logs UInt64/fromUnixTimestamp64Nano + protocol DateTime64(9)), the pipeline flowing, and the AE control-plane reproducibility loadtest. Sandbox-safe (no shell sleep; ClickHouse via kubectl exec). suite_test.go gains AELOAD_REPS for fast runs. Verified live on rig 6a58ec21: PASS=29 FAIL=0.
…onds schema_test.go: per-(table,column) fixtures asserting every timestamp column is nanosecond-precision — UInt64 unix-ns (kubescape_logs.event_time, watermark, dx_attack_graph.event_time) or DateTime64(9) (all protocol time_/event_time, adaptive_attribution t_start/t_end/last_seen, ae_reconcile, alerts). Plus TestNoCoarserTimestamps: dynamic backstop that fails on ANY DateTime coarser than (9) in forensic_db. evidence.sh runs both. Verified live: 37/37 columns PASS.
…bility tests are the suite)
TestJavaPocCalibration assumes the full stack (kubescape+vector+ClickHouse+ adaptive_export+dx+java-poc) and induces the java-poc listeriosis disease once, asserting each pipeline stage lights up as a before->after delta: 1 kubescape detects (R0001 spawn) 2 vector->ClickHouse carries it (fresh ns) 3 adaptive_export captures forensics (backend->pathogen:1389 LDAP egress) 4 dx rules in (non-blind verdict). Pathogen vocabulary (pathogen/disease/Specimen/listeriosis); external wire literal. Live+e2e gated (AELOAD_LIVE=1 AELOAD_E2E=1); sandbox-safe 3s polls, no long sleep. Nothing mocked — unlike the control-plane suite, the signal originates from a real workload and flows through every component.
This was referenced Jul 17, 2026
Merged
The calibration/e2e suite assumed a hand-installed stack (kubescape + vector + ClickHouse + the java-poc apps), which made runs non-reproducible. Add ONE command that stands the whole non-Pixie environment up, from each component's golden-source repo (one source of truth per component): - skaffold.yaml (module e2e-nonpixie): requires the soc stack Skaffold (ClickHouse -> kubescape -> vector) then the bob java-poc apps Skaffold (SBoBs -> workloads). Pixie itself (pl) is overlaid separately. - suite/deploy.go: EnsureE2EStack(t) invokes it (AELOAD_DEPLOY=1; AELOAD_CLEAN=1 for a from-scratch redeploy keeping pl + dx). TestJavaPocCalibration calls it, so the test deploys its own world when asked and otherwise assumes a live rig. Proven on a k3s rig: `skaffold deploy -m java-poc-apps -p k3s` applies namespaces -> SBoBs -> 5 workloads and waits them Ready. Refs k8sstormcenter/soc#231, k8sstormcenter/bob#152.
This was referenced Jul 17, 2026
Author
|
@build-agent — could I get a fresh AE image built from the current #68 HEAD ( Why: this is the last blocker for the e2e schema tests going green. The rig runs A ttl.sh image is fine. Thanks! |
Author
|
Scratch that — |
- confusion_test.go (TestStackEvidence): extracts per-workload KPIs (R0001/R0010, attribution, egress) across the whole java-poc stack and scores dx verdicts as a confusion matrix vs ground truth (only backend is malignant, only under log4shell). Reports always; asserts under AELOAD_MATRIX=1. Catches cross-scenario FPs — e.g. backend ruled_in [argocd-malicious-render] (a dx-repo issue) → FP=1, precision=0.50. - calibration_test.go stage4: poll the CURRENT backend pod's ruled_in (150s) instead of a saturated cumulative count that missed dx's workup lag; all-dx-pod aware. waitUntil now returns bool. - harness.go chReq: retry transient transport errors + 5xx (a kubectl port-forward EOF was flaking dedup-extend). Verified live on k3s: full suite green (schema + reproducibility std=0 + calibration); matrix report renders TP=1 FP=1 FN=0 TN=4.
entlein
added a commit
to k8sstormcenter/bob
that referenced
this pull request
Jul 17, 2026
… products seed) (#154) * fix(java-poc): real postgres-backed backend — #151 product-API image + products seed The golden java-poc source was captured from a rig running the pre-#151 STUB backend (@sha256:72655e...), whose ProductHandler returns {"products":[]} and never opens a postgres connection — and a users-only seed with no products table. Result: /api/products is empty, the observer's `curl -sf /api/products` fails, and there is no benign backend->postgres pg-wire on the normal path. Fix (both regressions): - backend image -> the #151 product-API build (@sha256:689e1c1b..., :latest / e47f8a8): real SELECT per /api/products, INSERT on checkout -> genuine backend->postgres connection. Verified live: /api/products returns the seeded products, /api/_probe = 200. - pg-seed -> the #151 products/users/orders schema+seed (was users-only). - LDAP_HOST=pathogen.pathogen-ns for the java-poc naming. Refs #151 (the product-API backend), #152 (golden source), k8sstormcenter/pixie#68. * feat(java-poc): add cleannoise benign noise-gen to the golden source cleannoise (busybox x3, wget frontend /api/products?q=noise) runs on every rig but was missing from the golden source — the confusion-matrix e2e test (TestStackEvidence) already lists it as a benign ground-truth workload (a true-negative), so a fresh skaffold deploy was incomplete without it. - 60-cleannoise.yaml — the deployment (captured from the live rig). - sbobs/cleannoise-nn.yaml — its user-defined NetworkNeighborhood (captured). - sbobs/cleannoise-ap.yaml — user-defined ApplicationProfile (authored: busybox sh/wget/sleep + the observer syscall set; the live AP had been wiped, so this is a reasonable hand-authored SBoB — regenerate from a learning run if needed). - skaffold.yaml — SBoBs before the workload, per the binding order. Refs pixie#68 (confusion_test.go references cleannoise), bob#152/#154. --------- Co-authored-by: Entlein <eineintlein@gmail.com>
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary: AE control surface gains bearer-JWT auth + TLS, the trigger gets a PollLimit-saturation escape, fork-cloud config restored after the main merge, and CodeRabbit Go-code findings on the umbrella PR #53 are all addressed (controller rollback, NaN-finite gate, MaxBytesReader, streaming retain-on-fail, etc.).
Test Plan: build + 14/14 AE go test packages green locally on cb81ecd / c2642da; CI-image lint (tools/linters/lint_like_ci.sh) reports 0 Errors 0 Warnings; vizier-release built aeprod22 (cb81ecd) successfully, aeprod23 (c2642da) in flight; trigger differential oracle added (oracle_test.go) pins watermark advancement + boundary dedup + PollLimit-at-capacity + the documented overflow trade-off.
Type of change: /kind feature
AE control auth + CodeRabbit #53 followup
Followup to #53 (merge #53 first — this stacks on
ae-prod; I'll retarget tomainonce #53 lands). Captured all 48 inline + 8 review + 12 issue CodeRabbit comments tobiz/PoC/log4j/ae_pr53_coderabbit_feedback/(JSON + DIGEST.md) so nothing is lost.✅ Done in this PR
px.dev/pixie/src/shared/services/utils);SetAuthverifies viajwtutils.ParseToken(signature + audience), middleware onHandler(),/healthzexempt. dx already mints this exact service JWT — it just attaches it. No new secret/crypto. Flag-gated (CONTROL_REQUIRE_AUTH, default off) for safe incremental rollout. Unit test mints a real JWT and asserts 401/pass/healthz. (CodeRabbit server.go: protect control endpoints.)t_end(≤0) and query windows (start≥end, non-positive). (server.go: reject invalid t_end/window.)Companion change (separate, entlein/dx)
dx
aeclient.gomust attachAuthorization: Bearer <minted JWT>on its AE control POSTs. Then flipCONTROL_REQUIRE_AUTH=true+ mountPL_JWT_SIGNING_KEYon AE. (Tracked; small.)Already resolved on ae-prod (verified — no action)
http_events=0"bug" — self-retracted by the author (misdiagnosis).Remaining AE-code Majors — triaged (to address as commits here)
Safe / clear:
streaming/filter.go:196— timer leak on deltaCh close.passthrough/passthrough.go:147— per-table query/write timeouts.pixieapi/pixieapi.go:89— direct-mode TLS-skip uses wrong option (→WithDirectTLSSkipVerify).stats.py:57— guard false "EXACT" when zero PASS rows.schema.sql:485— retention onae_reconcile(unbounded growth).controller.go:347— don't fan out Pixie rows when attribution persistence fails.Risky / needs careful testing before a presentation — defer with care:
controller.go:255— rehydrated active windows never restart fan-out (state-machine path).trigger/clickhouse.go:303— inclusive-watermark paging (this is the F8 watermark area — change only with the F8 fixtures).schema.sql:430—adaptive_attributionORDER BY key width (schema change — migration risk).sink/clickhouse.go:433—Recordblocks hot path on CH (async refactor — perf-sensitive).