Feat: MLflow operand controller for CR lifecycle and OTEL RBAC

[Bobbins228]

Summary

Adds MLflowOperandReconciler to the kagenti-operator, moving MLflow CR creation and OTEL collector RBAC bootstrap from the setup shell script into the operator. Also fixes several bugs in the OTel bootstrap that prevented traces from reaching MLflow on RHOAI clusters.

Changes

MLflow Operand Controller (original)

• internal/controller/mlflow_operand_controller.go (new) — MLflowOperandReconciler that watches DSC, creates MLflow CR, waits for readiness via requeue, creates OTEL RoleBindings in agent namespaces
• internal/controller/mlflow_operand_controller_test.go (new) — unit tests covering DSC absence, management state handling, MLflow CR creation, readiness checks, OTEL RoleBinding creation, ClusterRole fallback, and idempotency
• internal/mlflow/types.go — added MLflowSpec to MLflow type, added minimal DataScienceCluster types with v2 scheme registration
• cmd/main.go — registered MLflowOperandReconciler under --enable-mlflow gate
• internal/controller/mlflow_controller.go — updated kubebuilder RBAC markers for expanded mlflows verbs
• charts/kagenti-operator/templates/rbac/role.yaml — added datascienceclusters get/list/watch, expanded mlflows to create/update, added clusterroles get + bind (scoped to mlflow-operator roles)

OTel→MLflow Trace Export Fixes

• TLS verification — otel.go: Set ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt on the MLflow exporter so the collector trusts RHOAI's service-serving CA. Without this, exports fail with x509: certificate signed by unknown authority.
• Workspace header — Added mlflow.workspace Helm value (wired as --mlflow-workspace CLI flag). RHOAI requires x-mlflow-workspace header on every request; without it MLflow returns 500. The workspace maps 1:1 to a Kubernetes namespace and is now an explicit configuration input rather than a runtime discovery.
• Experiment ID — Added mlflow.experimentName Helm value (default kagenti-traces, wired as --mlflow-experiment-name). The bootstrap calls the MLflow API at startup to create or retrieve the experiment, then sets the correct x-mlflow-experiment-id header. Without a valid experiment ID, MLflow returns 422.
• ClusterRole preference — mlflow_operand_controller.go: Flipped resolveMLflowClusterRole to prefer mlflow-operator-mlflow-integration over mlflow-operator-mlflow-edit, ensuring gatewayendpoints/use permission is present.
• Removed hardcoded preset values — presets.go: Removed hardcoded x-mlflow-experiment-id: "0" and tls.insecure: true from YAML presets since these are now set programmatically.

Files changed

File	What
charts/kagenti-operator/values.yaml	Added mlflow.workspace and mlflow.experimentName
charts/kagenti-operator/templates/manager/manager.yaml	Wire new CLI flags from Helm values
kagenti-operator/cmd/main.go	Added --mlflow-workspace and --mlflow-experiment-name flags
kagenti-operator/internal/bootstrap/otel.go	TLS fix, config-driven workspace/experiment, MLflow API client for experiment creation
kagenti-operator/internal/bootstrap/otel_test.go	Updated tests for config-driven workspace, added workspace header test
kagenti-operator/internal/bootstrap/presets.go	Removed hardcoded experiment-id and insecure TLS
kagenti-operator/internal/controller/mlflow_operand_controller.go	ClusterRole preference fix

Manual Test Plan (OpenShift + RHOAI)

Prerequisites

• OpenShift cluster with RHOAI installed (MLflow operator available)
• kagenti-system namespace with kagenti-deps deployed
• Access to an LLM endpoint (vLLM or similar)

Step 1: Deploy the operator

helm upgrade --install kagenti-operator charts/kagenti-operator/ \
  -n kagenti-system \
  --set controllerManager.container.image.repository=<your-registry>/kagenti-operator \
  --set controllerManager.container.image.tag=<your-tag> \
  --set controllerManager.container.imagePullPolicy=Always \
  --set mlflow.enable=true \
  --set mlflow.workspace=team1 \
  --set mlflow.experimentName=kagenti-traces \
  --set otelBootstrap.enable=true

Step 2: Create agent namespace and LLM secret

kubectl create namespace team1

kubectl create secret generic llama-stack-inference-model-secret \
  --from-literal INFERENCE_MODEL="<model>" \
  --from-literal VLLM_URL="<url>" \
  --from-literal VLLM_TLS_VERIFY="<true|false>" \
  --from-literal VLLM_API_TOKEN="<token>" \
  -n team1

Step 3: Restart the operator (so bootstrap discovers the new namespace)

kubectl rollout restart deployment/kagenti-controller-manager -n kagenti-system
kubectl rollout status deployment/kagenti-controller-manager -n kagenti-system

Step 4: Verify bootstrap logs

kubectl logs deployment/kagenti-controller-manager -n kagenti-system | grep -E "bootstrap|MLflow|workspace|experiment"

Expected output:

• MLflow CRD detected, discovering MLflow CR
• Found available MLflow CR with correct tracesURL
• Created/found MLflow experiment with name: kagenti-traces, workspace: team1, and an experimentID
• OTel collector bootstrap complete

Step 5: Verify the OTel collector ConfigMap

kubectl get configmap otel-collector-config -n kagenti-system -o jsonpath='{.data.base\.yaml}'

Verify:

• x-mlflow-workspace: team1
• x-mlflow-experiment-id: "<id>" (matches bootstrap log)
• ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
• bearertokenauth/mlflow extension present

Step 6: Deploy a test agent (e.g. weather-service)

Deploy the weather agent and tool from agent-examples/ into team1, adjusting LLM env vars and removing runAsUser: 1000 from securityContext for OpenShift SCC compatibility.

Step 7: Send a request and verify traces

kubectl port-forward svc/weather-service -n team1 8080:8080 &

curl -s -X POST http://localhost:8080/ \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"message/send","id":"test-1","params":{"message":{"role":"user","messageId":"msg-1","parts":[{"type":"text","text":"What is the weather in London?"}]}}}'

Step 8: Confirm no export errors

kubectl logs deployment/otel-collector -n kagenti-system --since=5m | grep -iE "error|fail|reject|404|500|401"

Expected: No errors. Traces should flow cleanly to MLflow.

Test Results

• Unit tests pass (go test ./internal/bootstrap/ -v)
• go build ./... clean
• Pre-commit hooks pass (go-fmt, go-vet, helmlint)
• End-to-end tested on live OpenShift cluster with RHOAI MLflow
• Traces confirmed in MLflow workspace with correct experiment
• Zero export errors in OTel collector logs

[ChristianZaccaria]

Great work Mark!! I left two minor comments, after that we should move this from draft to open. Thanks!

What I tested:

1. Deployed operator with mlflow.enable=true, mlflow.workspace=team1, otelBootstrap.enable=true
2. Verified MLflow CR created in redhat-ods-applications namespace.
3. Deployed agent with kagenti.io/type=agent label in team1
4. Verified OTEL RoleBindings created in team1 and redhat-ods-applications
5. Verified OTEL collector ConfigMap contains workspace header, experiment-id, and ca_file
6. Confirmed traces reach MLflow with no 422/401 errors in collector logs
7. Deployed second agent in team2, confirmed RoleBinding auto-created

[ChristianZaccaria]

/lgtm thank you!

[cwiklik]

High-quality controller PR that moves MLflow CR creation + OTEL RBAC bootstrap out of the setup script and fixes three concrete RHOAI trace-export bugs (service-CA TLS trust, config-driven x-mlflow-workspace header instead of fragile hostname-parsing, programmatic experiment-id). Clean idiomatic controller-runtime: graceful CRD-absent handling, managementState-change predicate, requeue-based readiness via EndpointSlices. Excellent test coverage.

The two inline suggestions are about latent/upgrade correctness, not the happy path — neither blocks a fresh install.

Security: good — clusterroles/bind is scoped via resourceNames to the two MLflow roles; service-CA TLS replaces insecure: true; no secrets. Minor: clusterroles get is cluster-wide and could also be scoped to those two resourceNames.

Nits: newServiceCAHTTPClient ignores the AppendCertsFromPEM bool — a malformed service-ca.crt yields a silently empty CA pool and an opaque TLS failure later; consider erroring. Commit subjects use feat:/fix: without the emoji prefix Kagenti's convention favors (style only; DCO green). config/rbac/role.yaml doesn't appear to be regenerated alongside the marker changes (ties to comment #1).

All 15 CI checks passing.

[pdettori]

@Bobbins228 please rebase and consider addressing suggestions

[pdettori]

LGTM — approving.

All review feedback addressed:

1. Error logging (ChristianZaccaria) — ✅ log.Error used for experiment creation failure in otel.go
2. cert-manager RBAC (ChristianZaccaria) — deferred; Christian approved without it on Jun 8
3. RBAC markers (cwiklik) — ✅ markers now correctly reference discovery.k8s.io/endpointslices and scoped clusterroles get + bind
4. RoleBinding roleRef immutability (cwiklik) — ✅ delete+recreate logic added with dedicated upgrade-path test

CI status:

• PR title check: cosmetic (feat: → Feat: needed) — not a code issue
• E2E failure: pre-existing on main (Skill Discovery linkedSkills test) — unrelated to this PR; 35/36 E2E tests pass

Clean controller-runtime implementation with good test coverage. TLS fix and config-driven workspace/experiment are solid improvements over the previous hardcoded approach.

[pdettori]

@Bobbins228 I fixed the PR title as the CI checks requires "Feat" with capital f.