Skip to content

Fix: use correct MLflow API group and resource for agent RBAC#404

Merged
pdettori merged 1 commit into
rossoctl:mainfrom
rh-dnagornuks:fix-mlflow-integration-constants
Jun 10, 2026
Merged

Fix: use correct MLflow API group and resource for agent RBAC#404
pdettori merged 1 commit into
rossoctl:mainfrom
rh-dnagornuks:fix-mlflow-integration-constants

Conversation

@rh-dnagornuks

@rh-dnagornuks rh-dnagornuks commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the MLflow integration constants to use mlflow.kubeflow.org (API group) and experiments (resource) instead of the incorrect mlflow.opendatahub.io / mlflowexperiments. The MLflow gateway's Kubernetes authorization plugin performs SubjectAccessReview checks against mlflow.kubeflow.org/experiments, not the CRD's own API group — so agents were getting 403 PERMISSION_DENIED when exporting traces via the MLflow SDK.

This is a follow-up to #366. Further testing revealed that the Role permissions weren't properly configured, possibly because traces exported by the OTel collector masked failures in the agent's direct MLflow SDK path.

Changes

  • MLflowExperimentsAPIGroup: mlflow.opendatahub.iomlflow.kubeflow.org
  • MLflowExperimentsResource: mlflowexperimentsexperiments
  • Updated kubebuilder RBAC markers to match
  • Updated config/rbac/role.yaml and charts/kagenti-operator/templates/rbac/role.yaml
  • Updated docs/mlflow-integration.md to reflect correct values

Testing

  • Deployed an agent with MLFlow tracing enabled in code on a fresh ROSA cluster
  • Removed OTEL_EXPORTER_OTLP_ENDPOINT to isolate the MLflow SDK path
  • Verified POST /mlflow/api/3.0/mlflow/traces returns 200 with full span data (4 nested spans, token counts)

Made with Cursor

@cwiklik cwiklik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, well-scoped fix: the per-agent Role must grant on mlflow.kubeflow.org/experiments (what the MLflow gateway's authorization plugin checks via SubjectAccessReview), not the CRD's own mlflow.opendatahub.io/mlflowexperiments — which is why agents hit 403 on the direct MLflow SDK trace path.

Verified thorough:

  • Complete & consistent — updates the Go constants, the kubebuilder marker, both config/rbac/role.yaml and the chart role.yaml (correct 3-way RBAC sync), and the doc. The mlflows marker correctly stays on mlflow.opendatahub.io (the real CRD); only the gateway's virtual experiments resource moves to mlflow.kubeflow.org.
  • Regression-protectedmlflow_controller_test.go:244-245 asserts the generated Role uses the constants, so the corrected values are locked in (and explains the green Unit Tests without editing the test).
  • Empirically validated — author confirmed POST /mlflow/api/3.0/mlflow/traces returns 200 with full spans on a fresh ROSA cluster after isolating the SDK path.

CI — E2E red but unrelated: the one failure is SharedTrust E2E [BeforeAll] should create cacerts Secrets — timed out 120s (32/33 passed); an Istio shared-trust setup flake with no connection to MLflow RBAC constants. Recommend a re-run before merge.

Security: tightens correctness of least-privilege agent RBAC; scoped via resourceNames. No secrets.

Commits: signed-off. Nit: subject Fix: → conventional fix: (lowercase), lacks emoji prefix (style only).

@rh-dnagornuks rh-dnagornuks changed the title Fix: use correct MLflow API group and resource for agent RBAC fix: use correct MLflow API group and resource for agent RBAC Jun 8, 2026
@rh-dnagornuks

Copy link
Copy Markdown
Contributor Author

/run-e2e

@rh-dnagornuks
rh-dnagornuks force-pushed the fix-mlflow-integration-constants branch from 867758d to b8f2172 Compare June 10, 2026 08:44
@rh-dnagornuks rh-dnagornuks changed the title fix: use correct MLflow API group and resource for agent RBAC Fix: use correct MLflow API group and resource for agent RBAC Jun 10, 2026
@pdettori

Copy link
Copy Markdown
Member

@rh-dnagornuks CI has been fixed - please rebase from upstream main and force push so we can check now with full CI

The MLflow gateway's Kubernetes authorization plugin performs
SubjectAccessReview checks against mlflow.kubeflow.org/experiments,
not mlflow.opendatahub.io/mlflowexperiments. The scoped Role granted
permissions on a resource the gateway never checks, causing agents to
receive 403 PERMISSION_DENIED when calling the MLflow tracing API.

Signed-off-by: Daniels Nagornuks <dnagornu@redhat.com>

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
@rh-dnagornuks
rh-dnagornuks force-pushed the fix-mlflow-integration-constants branch from b8f2172 to 7353747 Compare June 10, 2026 19:29
@pdettori
pdettori merged commit fe7024d into rossoctl:main Jun 10, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants