Error: EACCES: permission denied in container on self hosted Linux runner

[mpconte]

In an effort to checkout a repo within a container that's being self hosted on a Linux VM running Ubuntu 20.04 as follows:

name: OS Build

# Controls when the workflow will run
on: 
    push:
      paths-ignore:
        - "Dockerfile"
        - ".github/workflows/docker_build.yml"
        - README.md
    pull_request:
    workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: camis-build-p01
    container:
      image: ghcr.io/sensoftinc/imx8mp_yocto_build_environment:1.0.0
      options: -u docker

With the image Docker file defined as:

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt update && apt upgrade -y && apt install ca-certificates -y && apt install wget locales -y && locale-gen en_US.UTF-8     
RUN apt install sudo
RUN apt install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath socat libsdl1.2-dev util-linux srecord -y
        
RUN apt install xterm sed cvs subversion coreutils texi2html docbook-utils python-pysqlite2 help2man make gcc g++ desktop-file-utils \
        libgl1-mesa-dev libglu1-mesa-dev mercurial autoconf automake groff curl lzop asciidoc -y

RUN apt install cpio python python3-pip python3-pexpect xz-utils debianutils iputils-ping \
        python3-git python3-jinja2 libegl1-mesa xsltproc fop dblatex xmlto pylint3 -y

RUN apt install u-boot-tools -y

RUN groupadd -r docker && useradd -r -g docker -ms /bin/bash -u 1001 docker && adduser docker sudo

I get the following error:

/usr/bin/docker exec  5b033937ed15061a8f606fa5f3805d0794caf9e04e3c12576fda15d25bde22ab sh -c "cat /etc/*release | grep ^ID"
node:internal/fs/utils:344
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_c7001c04-a974-4f62-8e53-a488[14](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:15)7475c5'
    at Object.openSync (node:fs:585:3)
    at Object.writeFileSync (node:fs:2[15](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:16)3:35)
    at Object.appendFileSync (node:fs:2215:6)
    at Object.issueFileCommand (/__w/_actions/actions/checkout/v3/dist/index.js:2293:8)
    at Object.saveState (/__w/_actions/actions/checkout/v3/dist/index.js:1[18](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:19)73:31)
    at Object.153 (/__w/_actions/actions/checkout/v3/dist/index.js:4044:10)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:[22](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:23):30)
    at Object.[28](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:29)7 (/__w/_actions/actions/checkout/v3/dist/index.js:7013:34)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:[30](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:31))
    at Object.853 (/__w/_actions/actions/checkout/v3/dist/index.js:[31](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3490287639/jobs/5841522655#step:3:32)801:36) {
  errno: -13,
  syscall: 'open',
  code: 'EACCES',
  path: '/__w/_temp/_runner_file_commands/save_state_c7001c04-a974-4f62-8e53-a488147475c5'
}

[nschmeller]

I'm getting this same error on GitHub-hosted runners, in the container docker.io/homebrew/ubuntu22.04:

Run actions/checkout@v3
/usr/bin/docker exec  ed6660d87643174caa84af01b2dbb9fdb674b0c924ad206c2a17f548d5f1eefb sh -c "cat /etc/*release | grep ^ID"
node:internal/fs/utils:344
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_07a163e4-5330-44dc-9944-6f61ac3f315f'
    at Object.openSync (node:fs:585:3)
    at Object.writeFileSync (node:fs:2153:35)
    at Object.appendFileSync (node:fs:2215:6)
    at Object.issueFileCommand (/__w/_actions/actions/checkout/v3/dist/index.js:2344:8)
    at Object.saveState (/__w/_actions/actions/checkout/v3/dist/index.js:11928:31)
    at Object.153 (/__w/_actions/actions/checkout/v3/dist/index.js:4095:10)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:30)
    at Object.287 (/__w/_actions/actions/checkout/v3/dist/index.js:7064:34)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:30)
    at Object.853 (/__w/_actions/actions/checkout/v3/dist/index.js:31838:36) {
  errno: -[13](https://github.com/nschmeller/dotfiles/actions/runs/3826496339/jobs/6510334541#step:3:14),
  syscall: 'open',
  code: 'EACCES',
  path: '/__w/_temp/_runner_file_commands/save_state_07a163e4-5330-44dc-9944-6f61ac3f3[15](https://github.com/nschmeller/dotfiles/actions/runs/3826496339/jobs/6510334541#step:3:16)f'
}

I'm invoking the action as

    runs-on: ubuntu-latest
    container:
      image: docker.io/homebrew/ubuntu22.04
    steps:
      - uses: actions/checkout@v3
        name: Clone this repository

[nschmeller]

Looks like #956 is related...

[nschmeller]

I think #956 has workarounds, so I think this issue can be closed.

The workaround that I used was to "override the default container user and use 'root'":

container: 
    image: alpine:latest
    options: --user root

[junaruga]

Thanks for the workaround.
I faced this kind of permission error on the "Post Run actions/check" process when running the container by a regular user.

https://github.com/junaruga/ruby/actions/runs/4175636293/jobs/7230829664

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_90003fcf-9614-4b4d-8680-bf040803c6fc'
    at Object.openSync (node:fs:585:3)
    at Object.writeFileSync (node:fs:2170:35)
    at Object.appendFileSync (node:fs:2232:6)
    at Object.issueFileCommand (/__w/_actions/actions/checkout

As an behavior of the software is a bit different between root and regular users, I still want to run the container by a regular user.

[Dev-Mus]

You can override the default container user using options: --user root

[junaruga]

Yes, that's what I am doing it now. But ideally, I want to run the unit tests in the container by a regular user. Because there is a bit of difference between running the program by a regular user and the root user in my case.

[Chocrates]

Adding my 2 cents as well, we want to specifically not run as root so the workaround doesn't work in our case. We figured any other way around this?

[rhomolka-drw]

I have my own workaround - nonroot:

# add to dockerfile
RUN mkdir -m 1777 /__w

[raganar-ironside]

+1
I am facing this similar issue after I upgraded my Github runner version from 2.303.0 to 2.308.0.

[Chocrates]

Today GitHub only supports root users on the container, so we likely will only get workarounds unless/until that changes.

[omri-shilton]

anyone has a working workaround?
my setup is that we are building our base image for running tests - that has all the packages installed.
the user in that base image needs to be non-root because of SQLAlchemy tests that require that.
we are running self hosted runner in K8S - runner-scale-set.

[AyushSehrawat]

Facing same issue on Ubuntu 22.04 LTS

[germa89]

I got rid of the EACCES after installing the runner in the / directory. So the path is /actions-runner.

I did create the directory using sudo, and then change the permisions and ownership using chown and chmod.

I hope it is useful for someone.

This is also related to #1552

[acbramley]

I get this when using a larger runner as well (following https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/running-jobs-on-larger-runners)

[joeyOBenchmark]

I have a self hosted actions-runner on Ubuntu 20.04. It runs without sudo. It was having this problem.

Adding this "cleanup old checkout" step is working for me.

steps:
      # The "cleanup old checkout" step is needed because of this bug: https://github.com/actions/checkout/issues/1014
      - name: cleanup old checkout  
        run: chmod +w -R ${GITHUB_WORKSPACE}; rm -rf ${GITHUB_WORKSPACE}/*;
      - name: Check out repository
        uses: actions/checkout@v4

[gnuton]

The easiest way to workaround this is to go back to actions/upload-artifact@v2 not ideal although but it works

[pavelslavinskiy]

pavel.slavinskiy@yandex.ru

[pavelslavinskiy]

Post job cleanup.
/usr/bin/git version
git version 2.43.0
Temporarily overriding HOME='/home/runner/work/_temp/82270d65-7fc0-4573-a3d3-808b3e966a08' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /home/runner/work/git-manpages-l10n/git-manpages-l10n
/usr/bin/git config --local --name-only --get-regexp core.sshCommand
/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
http.https://github.com/.extraheader

[pavelslavinskiy]

p

[pavelslavinskiy]

p

[betimcariad]

Hi all,

I am not sure if this is still a valid place to comment, but I got similar errors and I want to give you some feedback and workaround.

The initial idea is to not use root, so we wanted non-root access to some host folders inside our container.

The solution mentioned earlier RUN mkdir -m 1777 /__w does not work (anymore) as of today.

Runner version [v2.315.0] and [v2.316.0] were tested here on GHE 3.12.

Another hint I need to give:
https://docs.github.com/en/enterprise-server@3.12/actions/creating-actions/dockerfile-support-for-github-actions#user

Github still suggests to use root.

Anyways, this is the workaround for non-root access inside container.

1. We use --userns=host option, see: https://docs.docker.com/engine/security/userns-remap/#disable-namespace-remapping-for-a-container
2. Make sure that the UID/GID is the same on host and in the container. This means that we need to prepare the container with the correct UIDs/GIDs. This can be annoying if you have different runners with different UIDs/GIDs and you need to support all of them in your container. For example, in your Dockerfile you might need to add RUN useradd runner_1000 -m -u 1000 -s /bin/bash for the UID 1000 to exist inside the docker container.
3. You have to start now the container with the correct UID/GID, which is the same as on the host.
   How you provide it, is in your own discretion. We created an organization variable with a mapping of runners and usernames of containers with correct UID.

This made it work. The cleanup step was not throwing access errors anymore.

⚠️ Please do know that you might mitigate some security features and you might be exposing too many files and access rights.

I wish that Github would fix the access to the host folders that are anyways used inside the container...

Best regards

[cboettig]

Just to add, options: --user root does not appear to be a recognized option when using GitHub Action Runner Controller for self-hosting in kubernetes mode. (It is recognized in dind mode, which does not respect resource limits).

A better solution would be great.

[sebastienmoreno]

In my case the following rootless setup is working:

I setup a arc-runner-scale-set on "Ubuntu 22.04.5 LTS"

The values when installing the scale-set are:

containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "standard-rwo"
    resources:
      requests:
        storage: 5Gi

template:
  spec:
    securityContext:
      fsGroup: 1001
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER # This is required to run jobs without containers
          value: "false"

And my Docker image is using a 1001 user.

Note: I didn't see on the scale-set config how to force the user ID to another one, 1001 seems to be the ID a static configuration.