Dashboard is not showing - k8s

[dusansusic]

Zalenium Image Version(s):
3.12e
Docker Version:
17.3.2
OS:
ubuntu 16.04

I found an issue here: #353 and it aims that's solved but it's not.

When i deployed zalenium to the k8s, dashboard si not available till I log in into container:
kubectl exec -it zalenium-799b7bff48-45ktv bash -n infra
and execute:
sudo cp * videos/
it will work.

I am using k8s 1.10.3 accessing to dashboard:
http://IP:PORT/dashboard
or
http://IP:PORT/dashboard/ - problem exists on both ways

          volumeMounts:
            - name: zalenium-data
              mountPath: /home/seluser/videos

Should we use /home/seluser, instead of /home/seluser/videos?

Also, Service is not accessible at: http://IP:4444 but http://IP:PORT, eg. http://IP:32152

Service definition is identical to your:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zalenium
  name: zalenium
  namespace: infra
spec:
  ports:
  - name: zalenium
    port: 4444
    targetPort: 4444
  selector:
    app: zalenium
    role: grid
  type: NodePort

[dusansusic]

Anyone else who experience this issue with dashboard?

[pearj]

Port 4444 is the port that zalenium uses inside the container. Your nodeport is completely different, generally nodeports aren't useful for the internet anyhow. As most people would be using some sort of ingress.

On startup, it automatically tries to copy the dashboard files to the /home/seluser/videos mount. Maybe look for copy errors on zalenium pod start, maybe you have some sort of permissions issues on your volume mount.

[dusansusic]

@pearj Thanks for reply.

Well, ignore the parts with ports, my bad.

Volume is provisioned as EBS on AWS, I am not mounting it at all, k8s do that.
Container path is /home/seluser/videos and mount point on node is /tmp/videos.
I have permission issues:

Copying files for Dashboard...
cp: cannot create regular file '/home/seluser/videos/index.html': Permission denied
cp: cannot create directory '/home/seluser/videos/css': Permission denied
cp: cannot create directory '/home/seluser/videos/js': Permission denied

drwxr-xr-x 3 root root 4096 Jul 3 11:00 videos/

Script that copies this files should have chown -R seluser. /home/seluser/videos before coping files.

also, I have RBAC enabled cluster and I created roles:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zalenium-role
rules:
- apiGroups: [""]
  resources:
    - "pods"
  verbs: ["create", "get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zalenium-rbac
  namespace: infra
subjects:
- kind: ServiceAccount
  name: zalenium
  namespace: infra
roleRef:
  kind: ClusterRole
  name: zalenium-role
  apiGroup: rbac.authorization.k8s.io

then, I got this in logs:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default.svc/api/v1/namespaces/infra/pods/zalenium-40000-nkrnh. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "zalenium-40000-nkrnh" is forbidden: User "system:serviceaccount:infra:zalenium" cannot delete pods in the namespace "infra".

Why it needs permission to delete pods?

[dusansusic]

Created pull request #635 for changing this.

[pearj]

How did you configure your kubernetes cluster? It sounds broken to me, something seems broken with your ebs-provisioner. I've run an OpenShift cluster in AWS backed by EBS with each pod running without root privileges, and I've never needed to chown the volume before use.

[dusansusic]

provisioned with kops on aws, 3 masters + 3 nodes.

[pearj]

It needs to delete pods because once it has finished with a selenium container it deletes it unless you've configured it with maxTestSessions.

How come the ebs provisioner is mounting the volume at the node at /tmp/videos that seems strange to me. At least in my openshift all the volume mounts for a pod end up at something like /var/lib/minishift/openshift.local.volumes/pods/f34ef828-7ec4-11e8-abf3-3a16ff45e523/volumes/...
If it's using /tmp/videos, that sounds like you've hardcoded a host path and you're not using ebs at all...

[pearj]

I just left another comment on #635, I think you can just fix your problem with an init container. You could even use the same zalenium image since it needs to exist anyway, but just put the sudo chown line as the command: eg: command: ['sh', '-c', 'sudo chown -R seluser:seluser /home/seluser/videos']

[dusansusic]

Okay. Then, call it OpenShift example, not k8s. They're everything but not same product. Vanila k8s obviously has different behaviour.

[pearj]

Can you show me the zalenium PVC and related PV kubernetes objects? Also the zalenium pod object.

[dusansusic]

Sure, of course. Here are some info:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zalenium-data
  namespace: infra
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 80Gi

PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: zalenium-data
  namespace: infra
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 80Gi
  hostPath:
    path: "/tmp/videos"

deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: zalenium
  namespace: infra
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zalenium
        role: grid
    spec:
      containers:
        - name: zalenium
          image: dosel/zalenium:latest
          args:
            - start
            - '--desiredContainers'
            - '2'
            - '--screenWidth'
            - '1440'
            - '--screenHeight'
            - '810'
            - '--seleniumImageName'
            - 'elgalu/selenium'
            - '--maxTestSessions'
            - '4'
          env:
            - name: ZALENIUM_KUBERNETES_CPU_REQUEST
              value: 250m
            - name: ZALENIUM_KUBERNETES_CPU_LIMIT
              value: 500m
            - name: ZALENIUM_KUBERNETES_MEMORY_REQUEST
              value: 500Mi
            - name: ZALENIUM_KUBERNETES_MEMORY_LIMIT
              value: 1Gi
            - name: OVERRIDE_WAIT_TIME
              value: 5m
          resources:
            requests:
              cpu: 1000m
              memory: 1024Mi
          ports:
            - containerPort: 4444
              protocol: TCP
          volumeMounts:
            - name: zalenium-data
              mountPath: /home/seluser/videos
      volumes:
          - name: zalenium-data
            persistentVolumeClaim:
              claimName: zalenium-data
      serviceAccountName: zalenium

$ kubectl get pv,pvc -n infra | grep zalenium

persistentvolume/pvc-74925eb4-7e9f-11e8-b795-0a05c6d3a70a   80Gi       RWO            Delete           Bound       infra/zalenium-data                           gp2                      20h

persistentvolume/zalenium-data                              80Gi       RWO            Retain           Available                                                                          20h
persistentvolumeclaim/zalenium-data                           Bound     pvc-74925eb4-7e9f-11e8-b795-0a05c6d3a70a   80Gi       RWO            gp2            20h

ClusterRole and ClusterRoleBinding:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zalenium-role
rules:
- apiGroups: [""]
  resources:
    - "pods"
  verbs: ["create", "get", "list", "watch", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zalenium-rbac
  namespace: infra
subjects:
- kind: ServiceAccount
  name: zalenium
  namespace: infra
roleRef:
  kind: ClusterRole
  name: zalenium-role
  apiGroup: rbac.authorization.k8s.io

$ kubectl get po -n infra | grep zalenium

zalenium-40000-cmvl8                          1/1       Running   0          20h
zalenium-40000-j9g57                          1/1       Running   0          20h
zalenium-799b7bff48-9x8ps                     1/1       Running   0          20h

$ kubectl describe pod zalenium-799b7bff48-9x8ps -n infra

Name:           zalenium-799b7bff48-9x8ps
Namespace:      infra
Node:           masked_node_ip
Start Time:     Tue, 03 Jul 2018 11:11:58 +0200
Labels:         app=zalenium
                pod-template-hash=3556369904
                role=grid
Annotations:    <none>
Status:         Running
IP:             100.96.7.42
Controlled By:  ReplicaSet/zalenium-799b7bff48
Containers:
  zalenium:
    Container ID:  docker://88100ffb68e1086567e74a0519b6ce6b73b9e26c823586535bab2f58634a672e
    Image:         dosel/zalenium:latest
    Image ID:      docker-pullable://dosel/zalenium@sha256:02d30afcf3ab0a474039a230b79e03bdad48553ceb0ef98ce9ca53548164f952
    Port:          4444/TCP
    Host Port:     0/TCP
    Args:
      start
      --desiredContainers
      2
      --screenWidth
      1440
      --screenHeight
      810
      --seleniumImageName
      elgalu/selenium
      --maxTestSessions
      4
    State:          Running
      Started:      Tue, 03 Jul 2018 11:12:01 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1
      memory:  1Gi
    Environment:
      ZALENIUM_KUBERNETES_CPU_REQUEST:     250m
      ZALENIUM_KUBERNETES_CPU_LIMIT:       500m
      ZALENIUM_KUBERNETES_MEMORY_REQUEST:  500Mi
      ZALENIUM_KUBERNETES_MEMORY_LIMIT:    1Gi
      OVERRIDE_WAIT_TIME:                  5m
    Mounts:
      /home/seluser/videos from zalenium-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from zalenium-token-gm2rx (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  zalenium-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  zalenium-data
    ReadOnly:   false
  zalenium-token-gm2rx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  zalenium-token-gm2rx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

$ kubectl describe pod zalenium-40000-j9g57 -n infra

Name:         zalenium-40000-j9g57
Namespace:    infra
Node:         masked_node_ip
Start Time:   Tue, 03 Jul 2018 11:12:03 +0200
Labels:       app=zalenium
              createdBy=zalenium
Annotations:  <none>
Status:       Running
IP:           100.96.6.60
Containers:
  selenium-node:
    Container ID:   docker://cb4292c9aed28768369d22e62cbd7d3bfe3d7878dfb8b74fb9b687a7a5c86c12
    Image:          elgalu/selenium
    Image ID:       docker-pullable://elgalu/selenium@sha256:b33d2481874eb9544e12d2c08fa4136ef0c7cd64de5e93d23428b65168bf05a9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 03 Jul 2018 11:12:05 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  1Gi
    Requests:
      cpu:      250m
      memory:   500Mi
    Readiness:  exec [/bin/sh -c curl -s http://`getent hosts ${HOSTNAME} | awk '{ print $1 }'`:40000/wd/hub/status | jq .value.ready | grep true] delay=5s timeout=1s period=1s #success=1 #failure=60
    Environment:
      SEL_NODEPOLLING_MS:            106000
      SELENIUM_HUB_HOST:             100.96.7.42
      TZ:                            Europe/Berlin
      NOVNC_PORT:                    50000
      CHROME:                        false
      SELENIUM_MULTINODE_PORT:       40000
      WAIT_TIME_OUT_VIDEO_STOP:      20s
      FIREFOX:                       false
      SCREEN_HEIGHT:                 810
      WAIT_TIMEOUT:                  120s
      SCREEN_WIDTH:                  1440
      NOVNC:                         true
      SELENIUM_NODE_PARAMS:          
      SEND_ANONYMOUS_USAGE_INFO:     true
      GRID:                          false
      MULTINODE:                     true
      VIDEO_STOP_SLEEP_SECS:         1
      SELENIUM_NODE_REGISTER_CYCLE:  16000
      ZALENIUM:                      true
      SELENIUM_NODE_HOST:            {{CONTAINER_IP}}
      PICK_ALL_RANDOM_PORTS:         false
      BUILD_URL:                     
      VNC_PORT:                      60000
      SELENIUM_HUB_PORT:             4445
      SELENIUM_NODE_PROXY_PARAMS:    de.zalando.ep.zalenium.proxy.DockerSeleniumRemoteProxy
    Mounts:
      /dev/shm from dshm (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from zalenium-token-gm2rx (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  dshm:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  zalenium-token-gm2rx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  zalenium-token-gm2rx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

[pearj]

Ahh hostpath as expected:

hostPath:
    path: "/tmp/videos"

Oh, when you say volume is provisioned as EBS on AWS. Did you manually provision that? You aren't using the kubernetes dynamic volume provisioning?
Because if your cluster is configured for it, it can dynamically provision an EBS volume simply based on a PVC and then all the permissions will be setup correctly.

Even without the automatic provisioning, you can specify awsElasticBlockStore as a volume type and it will take care of the mounting for you, and I think formatting too.

https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore

[pearj]

If you're interested in enabling the EBS auto provisioning, you need to setup a storage class and potentially a few other things.

But this is a starting point:
https://kubernetes.io/docs/concepts/storage/storage-classes/#aws

Some other resources:
https://blog.thecodeteam.com/2017/06/05/use-dynamic-volume-provisioning-kubernetes/

Apparently, you need DefaultStorageClass enabled as an admission plugin too, not sure if kops does that by default or not.
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic

[dusansusic]

hm, to be honest I don't get it. When I say it's automatically provisioned by k8s, then it's provisioed by k8s. My cluster is provisioned by kops on AWS and when I am requesting resources, then nodes/masters are able to do that because of IAM roles.

I removed hostPath from Persistent Volume, PV is provisied as 80Gi EBS partition on AWS.

Namespace:     infra
StorageClass:  gp2
Status:        Bound
Volume:        pvc-59fd6c7b-80fb-11e8-9452-0259f683dc68
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      80Gi
Access Modes:  RWO
Events:
  Type    Reason                 Age   From                         Message
  ----    ------                 ----  ----                         -------
  Normal  ProvisioningSucceeded  31m   persistentvolume-controller  Successfully provisioned volume pvc-59fd6c7b-80fb-11e8-9452-0259f683dc68 using kubernetes.io/aws-ebs

and:
$ kubectl describe pv pvc-59fd6c7b-80fb-11e8-9452-0259f683dc68 -n infra

kubectl describe pv pvc-59fd6c7b-80fb-11e8-9452-0259f683dc68 -n infra
Name:            pvc-59fd6c7b-80fb-11e8-9452-0259f683dc68
Labels:          failure-domain.beta.kubernetes.io/region=us-west-2
                 failure-domain.beta.kubernetes.io/zone=us-west-2a
Annotations:     kubernetes.io/createdby=aws-ebs-dynamic-provisioner
                 pv.kubernetes.io/bound-by-controller=yes
                 pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    gp2
Status:          Bound
Claim:           infra/zalenium-data
Reclaim Policy:  Delete
Access Modes:    RWO
Capacity:        80Gi
Node Affinity:   <none>
Message:         
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://us-west-2a/vol-0d05ce00eec42f055
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

when I enter in Zalenium container:

seluser@zalenium-9f86cb856-t6mqd:~/videos$ ls
lost+found

and

Filesystem      Size  Used Avail Use% Mounted on
overlay          97G  6.3G   91G   7% /
tmpfs           3.8G     0  3.8G   0% /dev
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   97G  6.3G   91G   7% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/nvme1n1     79G   56M   75G   1% /home/seluser/videos
tmpfs           3.8G   12K  3.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.8G     0  3.8G   0% /sys/firmware

So, partition is mounted I would say. But permissions are still wrong.

[pearj]

That new pv is using auto provisioning now which is good. Not sure why it doesn’t mount the volume with open permissions. Are running zalenium in priveleged mode by any chance?

Anyway you can always using an init container to fix the permissions.

[matthew-horrocks]

I am having a similar problem.

The videos directory is mounted and files are copied. Is the group of the videos and logs directories correct?

seluser@zaleniumgrid-zalenium-hub-6ddcddf864-n9cwr:~$ df -h
Filesystem                                     Size  Used Avail Use% Mounted on
overlay                                         30G  5.4G   24G  19% /
tmpfs                                          1.7G     0  1.7G   0% /dev
tmpfs                                          1.7G     0  1.7G   0% /sys/fs/cgroup
/dev/sda1                                       30G  5.4G   24G  19% /etc/hosts
//netwealthkuber.file.core.windows.net/videos  1.0G     0  1.0G   0% /tmp/mounted
shm                                             64M     0   64M   0% /dev/shm
//netwealthkuber.file.core.windows.net/shared  1.0G  640K  1.0G   1% /home/seluser/videos
tmpfs                                          1.7G   12K  1.7G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                          1.7G     0  1.7G   0% /sys/firmware

seluser@zaleniumgrid-zalenium-hub-6ddcddf864-n9cwr:~$ ls -l
total 48276
-rwxrw-r-- 1 seluser root        4529 Jun 27 07:13 Analytics.md
drwxrwxr-x 1 seluser root        4096 Jun 27 07:16 css
-rwxrw-r-- 1 seluser root       13766 Jun 27 07:13 dashboard_template.html
-rwxrw-r-- 1 seluser root         959 Jun 27 07:13 index.html
drwxrwxr-x 1 seluser root        4096 Jun 27 07:16 js
-rwxrw-r-- 1 seluser root        2128 Jun 27 07:13 LICENSE.md
-rwxrw-r-- 1 seluser root        1761 Jun 27 07:13 list_template.html
-rwxrw-r-- 1 seluser root        1031 Jun 27 07:13 logback.xml
-rwxrw-r-- 1 seluser root        2738 Jun 27 07:13 logging_debug.properties
-rwxrw-r-- 1 seluser root        2699 Jun 27 07:13 logging_info.properties
drwxr-xr-x 2 seluser seluser     4096 Jul  6 12:56 logs
-rwxrwxr-x 1 seluser root        1287 Jun 27 07:13 start-browserstack.sh
-rwxrwxr-x 1 seluser root        2679 Jun 27 07:13 start-saucelabs.sh
-rwxrwxr-x 1 seluser root        1268 Jun 27 07:13 start-testingbot.sh
drwxrwxrwx 2 seluser seluser        0 Jul  6 12:53 videos
-rwxrwxr-x 1 seluser root         831 Jun 27 07:13 wait-browserstack.sh
-rwxrwxr-x 1 seluser root         983 Jun 27 07:13 wait-saucelabs.sh
-rwxrwxr-x 1 seluser root         770 Jun 27 07:13 wait-testingbot.sh
-rwxrw-r-- 1 seluser root       15086 Jun 27 07:13 zalando.ico
-rwxrw-r-- 1 seluser root    49293733 Jun 27 07:14 zalenium-3.12.0e.jar
-rwxrwxr-x 1 seluser root       33565 Jun 27 07:13 zalenium.sh

seluser@zaleniumgrid-zalenium-hub-6ddcddf864-n9cwr:~$ ls -l videos/
total 34
drwxrwxrwx 2 seluser seluser     0 Jul  6 12:56 css
-rwxrwxrwx 1 seluser seluser 15737 Jul  6 13:32 dashboard.html
-rwxrwxrwx 1 seluser seluser    46 Jul  6 13:32 executedTestsInfo.json
-rwxrwxrwx 1 seluser seluser   959 Jul  6 12:56 index.html
drwxrwxrwx 2 seluser seluser     0 Jul  6 12:56 js
-rwxrwxrwx 1 seluser seluser  1995 Jul  6 13:32 list.html
drwxrwxrwx 2 seluser seluser     0 Jul  6 13:32 logs
-rwxrwxrwx 1 seluser seluser 15086 Jun 27 07:13 zalando.ico
seluser@zaleniumgrid-zalenium-hub-6ddcddf864-n9cwr:~$

[dusansusic]

@pearj I think you're wrong, this with init container will not work because in the moment of executing init container nothing from zalenium container will exist. We need a line in script which will adjuct ownership. Sorry :)

@matthew-horrocks which k8s version are you using?

[matthew-horrocks]

@dusansusic 1.10.3 in Azure (AKS)

[matthew-horrocks]

I have a VM and k8s both running Zalenium. I've ensured that the k8s permissions on the videos directory and the files within exactly match the permissions on the VM, but the dashboard still won't load.

What logs are available? I'm not too familiar with how the routing etc. works for the grid / console / dashboard.

[pearj]

@dusansusic when you mount an empty volume to /home/seluser/videos nothing will exist in that directory anyway. So I’m not sure what you’re getting at. You would probably need to use the zalenium container as the init container so that when the chown command is run the right uid is used for seluser. Your fix in #635 relied on nothing from the zalenium container except the seluser user and group existing.

[pearj]

@matthew-horrocks the fact that files exist in the videos directory means that zalenium was able to copy the files over, so I think you are getting further than @dusansusic.

Also, it looks likes you’ve named your mounts backwards for what it’s worth, according to the df command you posted. Shared is mounted at videos and videos is mounted at /tmp/mounted.

[dusansusic]

Then I am making something on top (or below this). Problem exist, that's a fact and you cannot decline it.
If I change a deployment, then I can build my own container with this fix and then I don't have any reason to write this. The point is that k8s is mounting it on this way and that's not my fault. I reported that with hope we can merge the solution.
If you don't want to help, that's fine also.

[pearj]

I’m not saying that there isn’t a problem for you. All I’m saying is that your problem can be solved by an Init Container without the need changing the behaviour for everyone else.

I presume you have not yet tried the Init Container?
All you need to do is mount the same volume in the Init Container and the main container and any changes made in the Init Container will be available in the main container.

There is a working example of this here:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/#creating-a-pod-that-has-an-init-container

[matthew-horrocks]

yeah, I noticed the mounts, too.

I've redeployed the containers a few time since and the mounts have been correctly setup... I've not changed anything 😆

[pearj]

Thinking about it. I think I know the cause of your problem. Because zalenium by default runs as root and then switches to seluser (in OpenShift it never runs as root) then your implementation of kubernetes must restrict the permissions to the root user by default because it thinks that’s the user for the container.
So, I’d be open to a fix in zalenium.sh if it does a few extra checks before launching into chmod.
It’d like it to check if sudo is available and check if seluser is able to write to the videos directory. I’m pretty sure there is a bash test for that.

[dusansusic]

Tried with init but it says that seluser:seluser is not available.

[pearj]

Can you paste your zalenium pod please?

[pearj]

Did you use busybox for the init container? If so that would explain why it couldn’t find the user. In that case we’d need to find out what the uid/gid is since it wouldn’t know the same. Otherwise using zalenium as the init container will know about seluser.

I did a bit of research on the problem and it looks like it’s actually a pretty common problem with kubernetes: kubernetes/kubernetes#2630

So I apologise for thinking it was a cluster setup problem.

One of the solutions people are using is Init containers. Other potential solution is using supplemental groups, but it wasn’t clear to me if it was fully implemented or not. I’m not exactly sure how to reproduce it, as I don’t think minikube is affected.

[dusansusic]

@pearj I tried it with busybox. I tried dosel/zalenium:latest image. none is working.
with busybox, users are missing, with second directory is missing (what's expected):

chown: cannot access '/home/seluser/videos': No such file or directory

As I said, initContainer will not work.
If you cannot add chowning to this dir, I will stop here because this is consuming too much time setting up Zalenium, it's easier to fix it in the fly and get it working and do a stuff with Zalenium, not just with setup.

I would appreciate if you reconsider chowning videos folder before mounting. I didn't try this stuff with minikube or minishift because I am using real HA cluster where everything works but this.

[pearj]

@dusansusic it sounds like you didn’t actually mount the volume into the init container. Each container in the pod is allowed to mount any of the volumes, but you still need to specifically mount them.

All the volumes are configured before the init containers are started, so it should work once the syntax is right.

[dusansusic]

Works.

  replicas: 1
  template:
    metadata:
      labels:
        app: zalenium
        role: grid
    spec:
      initContainers:
        - name: zalenium-prepare
          image: dosel/zalenium:latest
          command: ['sh', '-c', 'sudo chown -R seluser:seluser /home/seluser/videos']
          volumeMounts:
            - name: zalenium-data
              mountPath: /home/seluser/videos```

[pearj]

Ahh good to know.

The following is more for my reference:
In theory, it might not even be required if we follow the approach of this comment:
kubernetes/kubernetes#2630 (comment)
Using securityContext: and fsGroup:, that should allow the volume provisioner to automatically grant extra permissions specified at fsGroup.

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#podsecuritycontext-v1-core

A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod: 1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume.

[matthew-horrocks]

OK, I'm confused.

In order for the dashboard to load, do the following files need to be in /home/seluser/videos?

seluser@zaleniumgrid-zalenium-hub-654c85f45c-h674z:~$ ls -l videos/
total 12
drwxr-xr-x 2 seluser seluser 4096 Jul 10 16:06 css
-rwxr-xr-x 1 seluser seluser  959 Jul 10 16:06 index.html
drwxr-xr-x 2 seluser seluser 4096 Jul 10 16:06 js

I've started up a k8s cluster and deployed Zalenium without mounting anything in /home/seluser/videos. I manually copied the files based on

zalenium/scripts/zalenium.sh

Lines 401 to 404 in af665ec

	echo "Copying files for Dashboard..."
	cp /home/seluser/index.html /home/seluser/videos/index.html
	cp -r /home/seluser/css /home/seluser/videos
	cp -r /home/seluser/js /home/seluser/videos

and the dashboard still does not load; I get the Whoops! The URL specified routes to this help page. console page displayed. This suggests to me that something else is wrong.

Does the /grid/dashboard route to the /home/seluser/videos directory or somewhere else?

[pearj]

@matthew-horrocks Ahh that means you're accessing zalenium on the wrong port, you must be hitting the selenium grid port instead of the nginx port.

Nginx is configured to serve the dashboard, not the selenuim grid as seen here:

zalenium/docker/nginx.conf

Lines 35 to 39 in af665ec

	location /dashboard/ {
	alias /home/seluser/videos/;
	include /etc/nginx/mime.types;
	index dashboard.html index.html;
	}

You have probably used the 4445 port in your kubernetes service, you need to use the 4444 port instead.

[diemol]

Based on this comment #631 (comment), I think the issue has been solved.

If any other issue pops up, please open a new one with detailed information.