Helm chart cannot select the AWS credential chain for S3 — Pod Identity / IRSA is unreachable

[spkane]

Component: deploy/charts/buzz (chart 0.1.7) + crates/buzz-relay/src/config.rs
Ref: main

Describe the bug
The relay supports resolving S3 credentials from the AWS credential chain, and the
workspace carries a dependency patch specifically so EKS Pod Identity works. But
that path cannot be selected through the Helm chart: a deployment that omits the S3
keys ends up using the literal placeholder credentials buzz_dev / buzz_dev_secret
and fails with opaque S3 403s instead of using the pod's IAM role.

Steps to reproduce
Install the chart in the production profile against real AWS S3, with a
secrets.existingSecret that omits BUZZ_S3_ACCESS_KEY / BUZZ_S3_SECRET_KEY, on a
pod whose service account has S3 access via EKS Pod Identity or IRSA.

• Expected: the relay resolves credentials from the pod's IAM role.
• Actual: it signs requests as buzz_dev and every media/git object operation fails
  with 403. On a default install the startup BUZZ_GIT_CONFORMANCE_PROBE makes this
  startup-fatal, so readiness never opens — with no log line indicating that placeholder
  credentials were substituted.

Expected behavior

Why it should work

Both storage constructors explicitly branch on empty credentials —
buzz-media/src/storage.rs::MediaStorage::new and
buzz-relay/src/api/git/store.rs::Store::new:

let creds = match (access_key.is_empty(), secret_key.is_empty()) {
    (false, false) => Credentials::new(Some(access_key), Some(secret_key), None, None, None),
    (true, true)   => Credentials::default(), // env, profile, web-identity/IRSA, container, IMDS
    _ => return Err(/* half-configured static deployment */),
};

…and the workspace Cargo.toml patches aws-creds to a fork precisely because the
crates.io version "cannot read EKS Pod Identity credentials
(AWS_CONTAINER_CREDENTIALS_FULL_URI + AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE),
which the relay pod on bb-block requires for S3 media + git storage."

So the capability is intended and shipped.

Why it doesn't

Two behaviors combine badly.

1. The chart renders the keys as optional secretKeyRefs (templates/deployment.yaml):

- name: BUZZ_S3_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: {{ include "buzz.envSecretName" . }}
      key: BUZZ_S3_ACCESS_KEY
      optional: true

If the key is absent from the Secret, optional: true means the env var is not set
at all — there is no chart value that renders it as an explicit empty string.

2. The relay substitutes non-empty placeholders for absent vars
   (crates/buzz-relay/src/config.rs):

s3_access_key: std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_else(|_| "buzz_dev".to_string()),
s3_secret_key: std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_else(|_| "buzz_dev_secret".to_string()),
"buzz_dev" is not empty, so MediaStorage::new takes the static branch with
placeholder credentials. Credentials::default() is never reached.

The chain branch is only reachable when the env vars are present and empty, since
std::env::var returns Ok("") for a set-but-empty var and unwrap_or_else doesn't fire.

Version and platform
Find your version at the bottom of the Settings sidebar. Write "unknown" if you can't determine it.

• Buzz version: unknown - helm chart 0.1.7
• OS: Linux

Logs / additional context

Suggested fix
Either half resolves it; both together is cleanest.

1. Relay — unwrap_or_else(|_| "buzz_dev") → unwrap_or_default(), so an absent var
   becomes empty and selects the chain. This also removes the silent-placeholder footgun.
   Local MinIO development would need those defaults moved into .env.example /
   compose rather than living in code.

2. Chart — add an explicit credential-source selector, e.g.
   s3.credentialSource: static | chain. On chain, emit BUZZ_S3_ACCESS_KEY: "" and
   BUZZ_S3_SECRET_KEY: "" as literal values instead of secretKeyRefs (or omit them
   entirely once the relay change lands). templates/_validate.tpl should then accept
   s3.bucket + credentialSource: chain as a valid S3 source, and hard-fail if
   chain is combined with a non-empty s3.accessKey.

No serviceAccount change is needed — the chart already creates a dedicated SA and
exposes serviceAccount.annotations (enough for IRSA's eks.amazonaws.com/role-arn),
and Pod Identity requires nothing on the SA because the association is AWS-side.

Workaround for anyone hitting this now
Have your secret manager emit literal empty strings for BUZZ_S3_ACCESS_KEY and
BUZZ_S3_SECRET_KEY (e.g. an External Secrets target.template.data block rather than a
remote-ref property). The secretKeyRefs resolve to "", and the relay takes the
credential-chain branch as intended.

Minor, related
The aws-creds fork pin is marked temporary pending durch/rust-s3#449. Until that
lands, Pod Identity support depends on an unmerged fork — worth noting in the chart README
so operators know what their Pod Identity support actually rests on.