When GitOps operator is run locally, it does not correctly setup the 'argoproj.io' Role rules for the 'argocd-application-controller' pod/service account.
When all of these are true:
- You run the GitOps operator locally, for example via
scripts/run_e2e_tests.sh or with make run-local
- You run it on a cluster that has not previously had the GitOps operator installed via OLM (and likewise not Argo CD operator)
You will find:
- That the
argocd-application-controller created within the openshift-gitops namespace is missing permissions to read/write argoproj.io CRs.
- That the following error occurs in the
argocd-application-controller pod log:
time="2021-06-28T19:46:10Z" level=error msg="Error persisting normalized application spec: applications.argoproj.io \"(application name)\" is forbidden: User \"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application- controller\" cannot patch resource \"applications\" in API group \"argoproj.io\" in the namespace \"openshift-gitops\"" application=(application name)
This is due to how the role is configured, as of argoproj-labs/argocd-operator#318.
When OLM installs an operator, it adds read/write access to the admin ClusterRole. However, if you don't use OLM to install the GitOps operator, then the admin cluster role will NOT contain the Argo CD CR permissions. Thus the admin cluster role cannot be guaranteed to contain the necessary cluster CRDs.
Unfortunately, as of argoproj-labs/argocd-operator#318, the admin ClusterRole is appended to the openshift-gitops-argocd-application-controller namespace Role. So if the admin ClusterRole is missing the Argo CD CRs (because the operator was not installed via OLM), then the openshift-gitops namespace will be missing permissions to write to Argo CD CRs.
TLDR: we can't rely on the admin ClusterRole to necessarily contain the rules we need, because it is written to by OLM, and not every CRD is installed via OLM.
To reproduce:
-
Start with a clean cluster (one on which no one has previously installed GitOps operator):
- To verify that you have a clean cluster, run the following command:
k get clusterrole/admin -o yaml | grep argoproj.io | wc -l
- It should return
0
-
cd (path to gitops-operator)
-
Run kubectl apply -f deploy/crds to install the CRDs (ignore the error 'The GitopsService "example-gitopsservice" is invalid')
-
Run make run-local
-
Wait for the openshift-gitops namespace to be created, and wait for Argo CD to be installed in it.
-
Run kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yaml
- Notice that it does NOT contain any rules for 'argoproj.io'
-
Delete the openshift-gitops namespace
-
Now install the latest OpenShift GitOp operator from OperatorHub
-
Wait for the openshift-gitops namespace to be created, and wait for Argo CD to be installed in it.
-
Run kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yaml
- Notice that it DOES contain rules that reference to 'argoproj.io'
The difference between steps 6 and 10 is the issue.
When GitOps operator is run locally, it does not correctly setup the 'argoproj.io' Role rules for the 'argocd-application-controller' pod/service account.
When all of these are true:
scripts/run_e2e_tests.shor withmake run-localYou will find:
argocd-application-controllercreated within theopenshift-gitopsnamespace is missing permissions to read/write argoproj.io CRs.argocd-application-controllerpod log:time="2021-06-28T19:46:10Z" level=error msg="Error persisting normalized application spec: applications.argoproj.io \"(application name)\" is forbidden: User \"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application- controller\" cannot patch resource \"applications\" in API group \"argoproj.io\" in the namespace \"openshift-gitops\"" application=(application name)This is due to how the role is configured, as of argoproj-labs/argocd-operator#318.
When OLM installs an operator, it adds read/write access to the
adminClusterRole. However, if you don't use OLM to install the GitOps operator, then theadmincluster role will NOT contain the Argo CD CR permissions. Thus theadmincluster role cannot be guaranteed to contain the necessary cluster CRDs.Unfortunately, as of argoproj-labs/argocd-operator#318, the
adminClusterRole is appended to theopenshift-gitops-argocd-application-controllernamespace Role. So if theadminClusterRole is missing the Argo CD CRs (because the operator was not installed via OLM), then the openshift-gitops namespace will be missing permissions to write to Argo CD CRs.TLDR: we can't rely on the
adminClusterRole to necessarily contain the rules we need, because it is written to by OLM, and not every CRD is installed via OLM.To reproduce:
Start with a clean cluster (one on which no one has previously installed GitOps operator):
k get clusterrole/admin -o yaml | grep argoproj.io | wc -l0cd (path to gitops-operator)Run
kubectl apply -f deploy/crdsto install the CRDs (ignore the error 'The GitopsService "example-gitopsservice" is invalid')Run
make run-localWait for the
openshift-gitopsnamespace to be created, and wait for Argo CD to be installed in it.Run
kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yamlDelete the openshift-gitops namespace
Now install the latest OpenShift GitOp operator from OperatorHub
Wait for the
openshift-gitopsnamespace to be created, and wait for Argo CD to be installed in it.Run
kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yamlThe difference between steps 6 and 10 is the issue.