feat(kernel): ctx-deadline cancellation, ABI/mTLS/CloudFetch controls, and log forwarding

[mani-mathur-arch]

What

Hardens the SEA-via-kernel backend across three areas — context cancellation, connection-security/CloudFetch controls, and log forwarding — all behind the cgo && databricks_kernel build tag. The default CGO_ENABLED=0 Thrift build is unchanged; every kernel-only option is rejected loudly on the Thrift path.

1. Honor context deadlines on connect and mid-fetch

Both blocking cgo calls now route through cancellable C-ABI entry points via a ctxWatcher that bridges a Go context onto a kernel cancel token:

• OpenSession → kernel_session_open_cancellable (connect)
• rows.go nextBatch → kernel_result_stream_next_batch_cancellable (mid-fetch)

Previously each blocked in an uninterruptible cgo call — a slow warehouse cold-start or a hung CloudFetch chunk ignored the caller's ctx deadline. The watcher fires the token on ctx.Done(), dropping the in-flight kernel request future (a real abort, not a detach). The call sites return the ctx error while preserving the kernel error as its cause, and evict a session-fatal connection before returning. A non-cancellable ctx yields a nil watcher → NULL token → the plain, unchanged path (zero overhead). CloseSession stays fire-and-forget.

Closes PECOBLR-3645 and the connect half of PECOBLR-3646.

2. ABI-version handshake + mTLS + CloudFetch toggle

Wires four new kernel C-ABI symbols onto the driver's experimental-config surface:

• ABI-version handshake — a sync.Once check at the top of OpenSession compares the linked library's kernel_abi_version() against the header's DATABRICKS_KERNEL_ABI_VERSION and refuses to connect on a mismatch, so a differently-built prebuilt archive can't be silently misread.
• WithKernelClientCertificate(cert, key) — mTLS client identity forwarded via the paired kernel_session_config_set_tls_client_certificate. Both PEM halves are required together; validateKernelConfig rejects an incomplete/unpaired credential loudly (via an explicit "configured" marker robust against nil-vs-empty) rather than failing open with no client identity. The private key is never logged.
• WithKernelCloudFetch(enabled) — a tri-state toggle (nil keeps the kernel default on; set forwards kernel_session_config_set_cloudfetch_enabled), distinct from the plain-bool WithCloudFetch whose unset state can't be told from false.

A reflective exhaustiveness guard ensures a future KernelExperimental field can't ship unforwarded/unrejected.

3. Forward kernel (Rust) logs into the driver logger

The kernel's internal Rust tracing events are routed into the same driver logger sink as the Go binding lines (including a custom logger.SetLogOutput) via a reverse-call callback (kernel_set_log_callback) — instead of going to stderr through kernel_init_logging. The one-knob log level (DATABRICKS_LOG_LEVEL / DBSQL_KERNEL_DEBUG + RUST_LOG) is mirrored into the kernel subscriber. The Go trampoline has a panic firewall (recover) and is nil-/wrong-type-safe; the forwarding sink snapshots the logger at first connect so forwarded events aren't re-gated by a later driver-level change.

Build

KERNEL_REV is pinned to the kernel revision that exports the cancellable, ABI-version, mTLS, CloudFetch, and log-callback symbols this driver code links against.

Tests

• Unit (tagged, in the kernel-backend CI job):
  • ctxWatcher — fires the token on cancel and on deadline, is nil-safe on an uncancellable ctx, and tears down cleanly without firing (TestCtxWatcher*).
  • mTLS/CloudFetch — option wiring, defensive PEM copy, incomplete-pair rejection, DeepCopy, and the Thrift-path rejection (kernel_config_test.go, kernel_experimental_test.go).
  • Log forwarding — level mapping, nil no-op, panic firewall, nil-/wrong-type-safe trampoline, and a regression test proving forwarded kernel events are not re-gated by the live driver level (TestLogSinkForward*, TestLogCallback*).
• Unit (untagged, CGO_ENABLED=0): the ABI-version compare (match + mismatch, TestCompareABI / TestABIVersionMatches), config classification/exhaustiveness guard, and pure config-assembly all run in the default build with no kernel lib linked.
• Staging e2e (kernel_e2e_test.go): connect under an already-cancelled ctx fails fast; a long streaming query past its deadline is cancelled promptly with context.DeadlineExceeded; query cancel mid-execution.

How we know it works: the tagged suite exercises the real cgo ctx→token bridge and the real callback round-trip (not mocks), the untagged suite guarantees the default build and its config contract are unaffected, and the staging e2e confirms the cancellation and connection-security behavior end-to-end against a live warehouse.

Co-authored-by: Isaac

[mani-mathur-arch]

Code Review Squad — Review

Score: 69/100 — HIGH RISK

The C-ABI/FFI mechanics are sound — security and language reviewers independently verified cert handling, memory ownership, cancel-token lifecycle, and the log trampoline against the kernel header at the pinned rev, both clean. Risk is concentrated in two test-coverage gaps (a dropped forwarding line or a broken connect-cancel path would ship green) and a UX/contract inconsistency where the standard WithCloudFetch(false) is silently ignored on the kernel path (flagged by 3 reviewers). No confirmed production correctness bug; the HIGH RISK band is driven by the test gaps, not by shipped defects.

---

Medium Findings

M2 — doc.go explicitly denies the headline mid-fetch-cancellation feature this PR ships (doc.go:275-277, unchanged by this diff)

This PR adds mid-fetch cancellation: rows.go nextBatch now calls kernel_result_stream_next_batch_cancellable with a ctx-driven cancel token (proven by the new TestKernelE2EQueryCancelDuringExecution). But the read-path cancellation paragraph in doc.go still asserts the opposite: "On the read path, context cancellation is honored at result-batch boundaries, not mid-fetch: an in-flight CloudFetch batch runs to completion before the cancel takes effect."

go doc github.com/databricks/databricks-sql-go on the merged commit would print a mid-fetch-cancel disclaimer directly contradicted by code in the same commit. Users read the stale doc, assume a hung CloudFetch GET is uninterruptible, and design around a limitation that no longer exists.

This finding has no in-diff line to anchor an inline comment (the paragraph sits just past the last doc.go hunk, which ends at line 266).

• Suggested fix: Replace the paragraph with the now-true behavior (mid-fetch cancel via the kernel cancel token; a NULL token on a non-cancellable ctx preserves the plain path).

[mani-mathur-arch]

Re M2 (from the review summary — doc.go read-path paragraph denied the mid-fetch cancellation this PR ships): fixed in 4137103. The paragraph now states that context cancellation is honored both at result-batch boundaries and mid-fetch (a cancel/deadline firing while a CloudFetch batch is in flight aborts that fetch), with the non-cancellable-context zero-overhead note. The stale "an in-flight CloudFetch batch runs to completion before the cancel takes effect" sentence is gone. (No inline thread to resolve — the paragraph sat past the last doc.go hunk.)

All 6 inline findings (#1–#6) are fixed and their threads resolved; details in each thread. Head is now 3c0a7c2, which also adds a hermetic mTLS/custom-CA/hostname-skip handshake test. CI is green.