Fix: use correct MLflow API group and resource for agent RBAC#404
Conversation
cwiklik
left a comment
There was a problem hiding this comment.
Correct, well-scoped fix: the per-agent Role must grant on mlflow.kubeflow.org/experiments (what the MLflow gateway's authorization plugin checks via SubjectAccessReview), not the CRD's own mlflow.opendatahub.io/mlflowexperiments — which is why agents hit 403 on the direct MLflow SDK trace path.
Verified thorough:
- Complete & consistent — updates the Go constants, the kubebuilder marker, both
config/rbac/role.yamland the chartrole.yaml(correct 3-way RBAC sync), and the doc. Themlflowsmarker correctly stays onmlflow.opendatahub.io(the real CRD); only the gateway's virtualexperimentsresource moves tomlflow.kubeflow.org. - Regression-protected —
mlflow_controller_test.go:244-245asserts the generated Role uses the constants, so the corrected values are locked in (and explains the green Unit Tests without editing the test). - Empirically validated — author confirmed
POST /mlflow/api/3.0/mlflow/tracesreturns 200 with full spans on a fresh ROSA cluster after isolating the SDK path.
CI — E2E red but unrelated: the one failure is SharedTrust E2E [BeforeAll] should create cacerts Secrets — timed out 120s (32/33 passed); an Istio shared-trust setup flake with no connection to MLflow RBAC constants. Recommend a re-run before merge.
Security: tightens correctness of least-privilege agent RBAC; scoped via resourceNames. No secrets.
Commits: signed-off. Nit: subject Fix: → conventional fix: (lowercase), lacks emoji prefix (style only).
|
/run-e2e |
867758d to
b8f2172
Compare
|
@rh-dnagornuks CI has been fixed - please rebase from upstream main and force push so we can check now with full CI |
The MLflow gateway's Kubernetes authorization plugin performs SubjectAccessReview checks against mlflow.kubeflow.org/experiments, not mlflow.opendatahub.io/mlflowexperiments. The scoped Role granted permissions on a resource the gateway never checks, causing agents to receive 403 PERMISSION_DENIED when calling the MLflow tracing API. Signed-off-by: Daniels Nagornuks <dnagornu@redhat.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
b8f2172 to
7353747
Compare
Summary
Fixes the MLflow integration constants to use
mlflow.kubeflow.org(API group) andexperiments(resource) instead of the incorrectmlflow.opendatahub.io/mlflowexperiments. The MLflow gateway's Kubernetes authorization plugin performs SubjectAccessReview checks againstmlflow.kubeflow.org/experiments, not the CRD's own API group — so agents were getting 403 PERMISSION_DENIED when exporting traces via the MLflow SDK.This is a follow-up to #366. Further testing revealed that the Role permissions weren't properly configured, possibly because traces exported by the OTel collector masked failures in the agent's direct MLflow SDK path.
Changes
MLflowExperimentsAPIGroup:mlflow.opendatahub.io→mlflow.kubeflow.orgMLflowExperimentsResource:mlflowexperiments→experimentsconfig/rbac/role.yamlandcharts/kagenti-operator/templates/rbac/role.yamldocs/mlflow-integration.mdto reflect correct valuesTesting
OTEL_EXPORTER_OTLP_ENDPOINTto isolate the MLflow SDK pathPOST /mlflow/api/3.0/mlflow/tracesreturns 200 with full span data (4 nested spans, token counts)Made with Cursor