Layers go missing between RUN statements

[chpatton013]

Expected behavior

Files created in a layer are visible during the next RUN statement.

Actual behavior

All FS changes from a layer are absent in the following RUN statement.

Information

• I have tried with the latest version of my channel (Stable or Edge)
• I have submitted Diagnostics
• I have included the Diagnostics ID in this Issue:
• Diagnostic ID: D944CBA1-6FB4-47BE-9682-E0B5F69CAD24/2018-04-13_18-21-38
• Windows Version: Win 10 Pro
• Docker for Windows Version: 18.03.0-ce

I posted this on the community forum first, but haven't seem any activity yet. I thought I would escalate to the devs, since I believe this to be a bug in Docker.

I'm trying to install msys2 using the Chocolatey package manager. My install command appears to complete successfully, but attempts to use installed files in subsequent RUN statements shows that changes from the install RUN statement are no longer present.

I've been able to reproduce this with a pretty small Dockerfile:

FROM microsoft/windowsservercore

ENV chocolateyUseWindowsCompression=false

RUN @powershell \
      -NoProfile \
      -ExecutionPolicy Bypass \
      -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" \
 && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin" \
 && md C:\tools \
 && type nul > C:\tools\checkpoint1

RUN type nul > C:\tools\checkpoint2 \
 && choco install msys2 --package-parameters="/NoUpdate" --confirm \
 && type nul > C:\tools\checkpoint3 \
 && dir C:\tools

RUN dir C:\tools

The first RUN statement installs Chocolatey, adds choco to the PATH, makes a directory, and adds a checkpoint file to that directory.

The second RUN statement makes a second checkpoint file, uses choco to install msys2, makes a third checkpoint file, and then prints the contents of our directory.

The third RUN statement repeats the print of our directory.

The attentive reader may expect the output of dir in the second and third RUN statements to be the same.

The actual result is that all files created in the second RUN statement are absent.

Exemplary output follows:

Step 1/5 : FROM microsoft/windowsservercore
 ---> ad6116672030
Step 2/5 : ENV chocolateyUseWindowsCompression=false
 ---> Using cache
 ---> 33b4c1cfe12f
Step 3/5 : RUN @powershell       -NoProfile       -ExecutionPolicy Bypass       -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"  && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"  && md C:\tools  && type nul > C:\tools\checkpoint1
 ---> Running in a027ec3b57c4
Getting latest version of the Chocolatey package for download.
Getting Chocolatey from https://chocolatey.org/api/v2/package/chocolatey/0.10.10.
Downloading 7-Zip commandline tool prior to extraction.
Extracting C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocInstall\chocolatey.zip to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocInstall...
Installing chocolatey on this machine
Creating ChocolateyInstall as an environment variable (targeting 'Machine')
  Setting ChocolateyInstall to 'C:\ProgramData\chocolatey'
WARNING: It's very likely you will need to close and reopen your shell
  before you can use choco.
Restricting write permissions to Administrators
We are setting up the Chocolatey package repository.
The packages themselves go to 'C:\ProgramData\chocolatey\lib'
  (i.e. C:\ProgramData\chocolatey\lib\yourPackageName).
A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin'
  and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.
Creating Chocolatey folders if they do not already exist.
WARNING: You can safely ignore errors related to missing log files when
  upgrading from a version of Chocolatey less than 0.9.9.
  'Batch file could not be found' is also safe to ignore.
  'The system cannot find the file specified' - also safe.
chocolatey.nupkg file not installed in lib.
 Attempting to locate it from bootstrapper.
PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding...
WARNING: Not setting tab completion: Profile file does not exist at
'C:\Users\ContainerAdministrator\Documents\WindowsPowerShell\Microsoft.PowerShe
ll_profile.ps1'.
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
 first prior to using choco.
Ensuring chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder
Removing intermediate container a027ec3b57c4
 ---> 5227d76fecc3
Step 4/5 : RUN type nul > C:\tools\checkpoint2  && choco install msys2 --package-parameters="/NoUpdate" --confirm  && type nul > C:\tools\checkpoint3  && dir C:\tools
 ---> Running in e23b859a6426
Chocolatey v0.10.10
Installing the following packages:
msys2
By installing you accept licenses for the packages.
Progress: Downloading msys2 20161025.0.0... 100%
Progress: Downloading chocolatey-core.extension 1.3.3... 100%
chocolatey-core.extension v1.3.3 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'
msys2 v20161025.0.0 [Approved]
msys2 package files install completed. Performing other installation steps.
Installing to: C:\tools\msys64
Extracting 64-bit C:\ProgramData\chocolatey\lib\msys2\tools\msys2-base-x86_64-20161025.tar.xz to C:\tools\msys64...
C:\tools\msys64
Extracting C:\tools\msys64\msys2-base-x86_64-20161025.tar to C:\tools\msys64...
C:\tools\msys64
Starting initialization via msys2_shell.cmd
PATH environment variable does not have C:\tools\msys64 in it. Adding...
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 The install of msys2 was successful.
  Software installed to 'C:\tools\msys64'
Chocolatey installed 2/2 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
 Volume in drive C has no label.
 Volume Serial Number is EC03-67FA
 Directory of C:\tools
04/12/2018  03:04 PM    <DIR>          .
04/12/2018  03:04 PM    <DIR>          ..
04/12/2018  03:04 PM                 0 checkpoint1
04/12/2018  03:04 PM                 0 checkpoint2
04/12/2018  03:04 PM                 0 checkpoint3
04/12/2018  03:04 PM    <DIR>          msys64
               3 File(s)              0 bytes
               3 Dir(s)  135,626,526,720 bytes free
Removing intermediate container e23b859a6426
 ---> 7ea4e6a0433c
Step 5/5 : RUN dir C:\tools
 ---> Running in be3bde0c28e2
 Volume in drive C has no label.
 Volume Serial Number is EC03-67FA
 Directory of C:\tools
04/12/2018  03:04 PM    <DIR>          .
04/12/2018  03:04 PM    <DIR>          ..
04/12/2018  03:04 PM                 0 checkpoint1
               1 File(s)              0 bytes
               2 Dir(s)  136,087,236,608 bytes free
Removing intermediate container be3bde0c28e2
 ---> 9302aac014ec
Successfully built 9302aac014ec

I've spent several days looking for existing posts or bug reports for this problem, but so far haven't had any luck. Just coming up with a succinct description of the problem is hard enough, let alone trying to google for "docker missing layers".

I'm happy to provide whatever additional information I can to help get to the bottom of this.

Steps to reproduce the behavior

Nothing special. Just docker build the above Dockerfile.

[thaJeztah]

ping @johnstep @jhowardmsft

[Ranchester]

Hi,

I'm having the exact same issue with Docker 18.03 but in my case it happens when trying to install cygwin and cyg-get. This used to work with Docker 17.12.0.

When inspecting image history this can be seen:

Dockerfile:

FROM microsoft/windowsservercore
...
...
RUN powershell.exe -Command "iwr https://chocolatey.org/install.ps1 -UseBasicParsing | iex"
RUN powershell.exe -Command "refreshenv"
RUN powershell.exe -Command "choco install cyg-get -y --verbose"
...

Image history:

17.12.0

...
                                         2 weeks ago         cmd /S /C powershell.exe -Command "choco install cyg-get -y --verbose"
                                 222MB
...

18.03

...
  23 hours ago        cmd /S /C powershell.exe -Command "choco install cyg-get -y --verbose"
                                 393kB
...

So we can clearly see by comparing the sizes that something is wrong with 18.03 of handling this layer. Tried this also with cygwin installer without chocolate. Same results.
I'm about to test this also with the newest edge release if it's fixed already.

br.
Ranchester

[lowenna]

Looking

[lowenna]

It's something between c11508a and 6ed1163 in the moby/moby repo. Figuring out which commit

[lowenna]

Huh. It's moby/moby#35924. Will figure out why that makes a difference.

[lowenna]

Figured it. A simple fix.

[lowenna]

microsoft/hcsshim#165

[lowenna]

moby/moby#36938 is the fix. Needs back-porting to CE/EE.

[mback2k]

This may also be related to the following issue: moby/moby#36919

[stephen-turner]

I'm going to close this issue now as it's been identified as an upstream bug. It can be tracked on the upstream ticket.

[chpatton013]

Thanks for sorting this out so quickly!

[rsherk]

Hi,

Hopefully someone here can help me figure out when this will make it into production. Not sure if I should expect a fix in Docker or in the Windows images or somewhere else? Maybe I'm missing something. I've been banging my head against this for days...

I am seeing this with both stable: 18.03.1-ce-win65 (17513) and edge: 18.05.0-ce-win67 (18263) using microsoft/windowsservercore (1709, 1803, latest). The layer with cygwin never makes it into the image.

Let me know if this is the wrong forum for this.

[mback2k]

Unfortunately an update of hcsshim to v0.6.10 (okay) or v0.6.11 (better) has not made it into docker-ce 18.03 or 18.05 yet. Both versions are currently stuck on the broken commit microsoft/hcsshim@79062a5 which does not include microsoft/hcsshim@8fccb44. 😿

It would be really nice if 18.06 could include at least hcsshim v0.6.11.

[thaJeztah]

Yes; that fix will be in 18.06

[mback2k]

@thaJeztah Thanks a lot. Can you give us an ETA for a release of 18.06 and are there also any plans to provide an update to the preview version 17.10.0-ee-preview-3 for Windows Server 1709? I would really appreciate an answer, thank you very much in advance. :-)

[rsherk]

@thaJeztah @mback2k Thanks for the info and clarification. Much appreciated.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked