Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Document required cluster permissions to create CRDs #269

Open
ckadner opened this issue Nov 30, 2021 · 1 comment
Open

Document required cluster permissions to create CRDs #269

ckadner opened this issue Nov 30, 2021 · 1 comment

Comments

@ckadner
Copy link
Member

ckadner commented Nov 30, 2021

Describe the bug

When trying to "Launch" a Dataset which attempts to create a PVC, users may get a 403 error, since, by default Kubeflow cannot deploy any CRD resource on the cluster.

To Reproduce

Steps to reproduce the behavior:

  1. Go to 'Datasets' and select a Dataset
  2. Select the "Launch" tab
  3. Fill in a "Namespace", i.e. kubeflow or mlx
  4. Click "Submit"

Expected behavior

The Kubeflow Pipeline Run graph should appear.

Required fix for the user

  • the user / admin need to patch the cluster:
    kubectl create clusterrolebinding pipeline-runner-extend --clusterrole cluster-admin --serviceaccount=kubeflow:pipeline-runner
    

Required changes in MLX

Either ...

  1. Add the clusterrolebinding to all manifests (multi-user excluded?)
  2. Add it to each of the MLX deployment READMEs

Also add troubleshooting steps on the Dataset README

@yhwang @Tomcli

@yhwang
Copy link
Member

yhwang commented Nov 30, 2021

you would see log messages similar to this:

kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '29fea5a9-ee25-40ef-a65b-e67858904c70', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'd88dd123-8da1-4d61-9dd7-37f7dd5f43dd', 'Date': 'Tue, 30 Nov 2021 17:31:33 GMT', 'Content-Length': '364'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"datasets.com.ie.ibm.hpsys is forbidden: User \"system:serviceaccount:kubeflow:pipeline-runner\" cannot list resource \"datasets\" in API group \"com.ie.ibm.hpsys\" in the namespace \"kubeflow\"","reason":"Forbidden","details":{"group":"com.ie.ibm.hpsys","kind":"datasets"},"code":403}

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants