Operation not permitted - changing permission

[crossbill]

New to ansible-bender, was trying to create a custom awx_task/awx_web container using ansible-bender

host os: ubuntu
Default installation: haven't changed any config

Am I seeing the following error message, since the original container was build as root user?

Error message:

DEBUG  Using module file /usr/local/lib/python3.6/dist-packages/ansible/modules/system/setup.py
DEBUG  <new_container-20190530-151634637425-cont> PUT /home/ubuntu/.ansible/tmp/ansible-local-14723e1idq7f/tmpmjzuje4_ TO /tmp/ansible-tmp-1559229413.680333-66566591349185/AnsiballZ_setup.py
DEBUG  <new_container-20190530-151634637425-cont> RUN [b'buildah', b'run', b'--', b'new_container-20190530-151634637425-cont', b'/bin/sh', b'-c', b'chmod u+x /tmp/ansible-tmp-1559229413.680333-66566591349185/ /tmp/ansible-tmp-1559229413.680333-66566591349185/AnsiballZ_setup.py && sleep 0']
DEBUG  STDOUT b"chmod: changing permissions of '/tmp/ansible-tmp-1559229413.680333-66566591349185/AnsiballZ_setup.py': Operation not permitted\nerror while running runtime: exit status 1\n" STDERR b"chmod: changing permissions of '/tmp/ansible-tmp-1559229413.680333-66566591349185/AnsiballZ_setup.py': Operation not permitted\nerror while running runtime: exit status 1\n"
DEBUG  <new_container-20190530-151634637425-cont> RUN [b'buildah', b'umount', b'--', b'new_container-20190530-151634637425-cont']
DEBUG  RC 0 STDOUT b'' STDERR b''
DEBUG  fatal: [new_container-20190530-151634637425-cont]: FAILED! => {
DEBUG      "msg": "Failed to set execute bit on remote files (rc: 1, err: chmod: changing permissions of '/tmp/ansible-tmp-1559229413.680333-66566591349185/AnsiballZ_setup.py': Operation not permitted\nerror while running runtime: exit status 1\n)"
DEBUG  }
DEBUG
DEBUG  PLAY RECAP ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
DEBUG  new_container-20190530-151634637425-cont : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
DEBUG

Playbook:

- name: ansible-bender functionality
  hosts: all
  vars:
    ansible_bender:
      base_image: docker.io/ansible/awx_web

      target_image:
        name: new_container
        labels:
          built-by: '{{ ansible_user }}'


  tasks:
  - name: Run a sample command
    command: 'ls -al'

[TomasTomecek]

I can reproduce, seems that buildah doesn't like that image:

PLAY [ansible-bender functionality] *****************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
fatal: [new_container-20190531-100731782857-cont]: FAILED! => {"msg": "Failed to set execute bit on remote files (rc: 1, err: chmod: changing permissions of '/tmp/ansible-tm
p-1559290180.3841994-199304213862925/AnsiballZ_setup.py': Operation not permitted\nerror while running runtime: exit status 1\n)"}

PLAY RECAP ******************************************************************************************************************************************************************
new_container-20190531-100731782857-cont : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

But I when I try to play with the image, it works fine:

$ buildah from docker.io/ansible/awx_web
awx_web-working-container

$ buildah run -t awx_web-working-container -- bash
bash-4.2$ cd /tmp
bash-4.2$ ls
bash-4.2$ id
uid=1000 gid=0(root) groups=0(root)
bash-4.2$ touch /tmp/asd
bash-4.2$ chmod 0000 /tmp/asd
bash-4.2$ chmod 0777 /tmp/asd

I don't understand what's wrong.

To make things worse, the same error happens when I build as root. So this is definitely a bug, but I literally have no idea if this is bender, ansible or buildah.

[crossbill]

I have also noticed the same, can seen the error even when run as root. I was at the impression that it was buildah, since bender was trying to run a buildah command to change permission on the folder?

[TomasTomecek]

since bender was trying to run a buildah command to change permission on the folder?

It's not bender, it's ansible who copies the AnsiballZ_setup.py file to /tmp in the container, then tries to change the perms and then invoke the file.

[crossbill]

Seeing a similar error message in a issue raised in splunk/docker-splunk#105

Wondering if this an issues with the storage driver. I recon buildah defaults to overlay and doesn't support overlay2.

[laseryuan]

To make things worse, the same error happens when I build as root. So this is definitely a bug, but I literally have no idea if this is bender, ansible or buildah.

I get around it by building a base image. Switching the user to root, Dockerfile :

FROM ansible/awx_web
USER root

[daraul]

To add to the confusion: I get this error while building a docker image with packer, but only if I don't use root. My molecule playbooks work just fine on the same base image (ubuntu:18.04). The prepared image's HCL file is below if you're interested.

packer {
  required_plugins {
    docker = {
      version = ">= 0.0.7"
      source = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "ubuntu" {
  image  = "ubuntu:18.04"
  commit = true
  changes = [
    "WORKDIR /home/esd",
    "USER esd"
  ]
  run_command = ["-d", "-i", "-t", "{{.Image}}", "/bin/bash"]
}

build {
  name    = "ubuntu"

  source "source.docker.ubuntu" {
    name = "development"
  }

  provisioner "shell" {
    # Install some requirements that the ubuntu container doesn't come with.
    inline = [
      "apt-get update && apt-get install -y python3.6 acl",
      "add-apt-repository 'deb http://archive.ubuntu.com/ubuntu/ bionic universe' && apt-get update",
      "apt-get install -y python3.6 python3-pip python-urllib3 python-openssl python-pyasn1 sudo",
      "useradd --groups sudo -ms /bin/bash esd",
      "echo esd:redacted | chpasswd",
    ]
  }

  post-processors {
    post-processor "docker-tag" {
      repository = "frappe-base"
      tags = ["latest"]
    }

    post-processor "docker-save" {
      path = "prepared.tar"
    }
  }
}

The playbook below attempts to run some configuration on the "prepared" image above, with the esd user. It consistently fails at "Gathering Facts".

packer {
  required_plugins {
    docker = {
      version = ">= 0.0.7"
      source = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "ubuntu" {
  image  = "frappe-base:latest"
  pull = false
  commit = true
  run_command = ["-d", "-i", "-t", "{{.Image}}", "/bin/bash"]
}

build {
  name    = "ubuntu"

  source "source.docker.ubuntu" {
    name = "development"
  }

  provisioner "ansible" {
    playbook_file = "./frappe-bench.yml"
    user = "esd"
  }

  post-processors {
    post-processor "docker-tag" {
      repository = "frappe"
      tags = ["latest"]
    }

    post-processor "docker-save" {
      path = "docker.tar"
    }
  }
}