Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) on Docker Desktop for Windows Community edition 2.3.0.3(45519) #835

Open
cosmo0920 opened this issue Jun 5, 2020 · 33 comments
Open

hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) on Docker Desktop for Windows Community edition 2.3.0.3(45519) #835

cosmo0920 opened this issue Jun 5, 2020 · 33 comments

Comments

@cosmo0920
Copy link

I am/we are encountering "hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)" on Windows 10 Pro 1909 with Docker Community 19.03.8:

re-exec error: exit status 1: output: time="2020-06-05T14:05:25+09:00" level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" importFolderPath="C:\\ProgramData\\Docker\\tmp\\hcs949648159" path="\\\\?\\C:\\ProgramData\\Docker\\windowsfilter\\3a7d2d6226be0f3cca2d17d5b29069918b0f14dd889484ecd8bb208329d382b8"
hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)

> docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:37:20 2020
  OS/Arch:          windows/amd64
  Experimental:     false

It happens during msys2 installation of the minimal reproducible Dockerfile:

# Reproducible (0x3) error docker file

FROM mcr.microsoft.com/windows/servercore:ltsc2019

# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
RUN powershell -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"

# Fluentd depends on cool.io whose fat gem is only available for Ruby < 2.5, so need to specify --platform ruby when install Ruby > 2.5 and install msys2 to get dev tools
RUN choco install -y ruby --version 2.6.5.1 --params "'/InstallDir:C:\ruby26'" \

This issue is also happen with using mcr.microsoft.com/windows/servercore:1909 and mcr.microsoft.com/windows/servercore:1903 base images.

We also hit this issue with this Dockerfile.

And I didn't solve this issue with using https://github.com/jhowardmsft/docker-ci-zap and delete all docker related data.

How should we avoid this issue?
This was referenced Jun 5, 2020
Closed
Closed
@cosmo0920
Copy link
Author

cosmo0920 commented Jun 8, 2020 •
edited
Loading

I'd read the similar issues:

But they didn't help us. Our situation is freshly installed and docker-ci-zap execution didn't help....

@schribl
Copy link

schribl commented Jun 8, 2020

I am experiencing exactly the same issue as @cosmo0920 when trying to create a docker container that uses choco install msys2. The docker version also matches. It runs on Windows 10 Enterprise 1809 and the base images is mcr.microsoft.com/dotnet/framework/runtime:4.8

@cosmo0920
Copy link
Author

cosmo0920 commented Jun 9, 2020 •
edited
Loading

This issue also happens on edge channel.

  • Docker Desktop Community 2.3.1.0 Windows containers

(added) This issue also happens on Windows 10 Pro 2004 with Docker Desktop for Windows Community edition 2.3.0.3(45519).

@jeffreycoe
Copy link

I'm experiencing this exact issue on a Windows Server 2019 (10.0.17763) instance running Docker 19.03.5, too.

@farzonl
Copy link

farzonl commented Jun 19, 2020

I am also seeing this issue

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR C:\
COPY Windows/choco.ps1 choco.ps1
RUN .\choco.ps1

and the contents of the choco.ps1 is

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

choco install cygwin wget -y

@kempniu
Copy link

kempniu commented Jul 1, 2020

I am experiencing the same problem with Windows Server 2016 as the host system, Docker 19.03.5 (which is the latest version currently available for that system through DockerMsftProvider), and microsoft/dotnet-framework:3.5-sdk-windowsservercore-ltsc2016 as the base image.

In my case, installing cygwin using choco install triggers a hcsshim::ImportLayer error during docker build. When run in a "live" container (created using docker run), the same command works fine, but subsequently running docker commit for such a container triggers the exact same error. The Dockerfile I am using was working fine back in January 2020 with the same Docker version.

I tried using docker-ci-zap and starting the Docker daemon with --storage-opt size=50G to no avail.

I would be happy to do any testing that could help resolve this issue as it is currently preventing me from provisioning any new CI server.

@QShen3
Copy link

QShen3 commented Jul 3, 2020

The same as @kempniu

Re-install the Docker and re-pull the base image can not solve this problem.

@SergeyMN
Copy link

SergeyMN commented Aug 4, 2020

Not sure if this related in some way or not but I have the issue on windows server 2019 with
level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)"
whenever I start sysmon (particularly sysmondrv) until it is unloaded or uninstalled.

@eliasfilipec
Copy link

I had this problem and solved it with the following steps:
1 ° I executed the following commands:
to clear the images:
docker images prune
2 ° I executed the one that cleans everything (Images, Containers, Cache and Networks)
docker system prune

@woernsn
Copy link

woernsn commented Aug 27, 2020

Same problem for me.
The problem seems to be cygwin.

Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:43:18 2020
OS/Arch: windows/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.24)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:57:30 2020
OS/Arch: windows/amd64
Experimental: false

@bryce-shang bryce-shang mentioned this issue Oct 1, 2020
Merged
@chvjak
Copy link

chvjak commented Oct 21, 2020

@kempniu @farzonl @schribl I run into same problem trying build an image with Cygwin. It seems that it is hardlinks which Cygwin uses a lot are not handled correctly by Docker. After I got rid of the hardlinks in Cygwin installation I was able to commit the image without problems. To get rid of the hardlinks I have zipped and unzipped Cygwin folder.

@rolyp rolyp mentioned this issue Dec 14, 2020
Closed
27 tasks
@jaunruh
Copy link

jaunruh commented Feb 9, 2021

I am encountering the same issue. I am using the mcr.microsoft.com/dotnet/framework/aspnet:4.8 container in a Dockerfile. When building using this Dockerfile I get the error level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3). The error is thrown on some step that downloads some larger files. BUT when using powershell inside of the container to download the files, I do not encounter the issue.

@directhex
Copy link
Member

For the Cygwin folks, my workaround looks like this:

RUN setx /M CYGWIN "winsymlinks:lnk"

RUN curl -SL --output %TEMP%\cygsetup.exe https://cygwin.com/setup-x86_64.exe `
    && powershell -Command `
        Start-Process %TEMP%\cygsetup.exe -Wait -ArgumentList '--quiet-mode --wait --root c:\cygwin --local-package-dir c:\cygwin --site http://mirrors.kernel.org/sourceware/cygwin/ --packages autoconf' `
    && C:\cygwin\bin\bash.exe -c 'for i in `/bin/find / -xdev -type l`; do j=`/bin/readlink $i`; /bin/rm -f $i; /bin/ln -sf $j $i; done' `
    && powershell -Command `
        Remove-Item -Recurse -Force c:\cygwin\http*

This forces Cygwin to use legacy (.lnk) based links, then recursively goes through Cygwin's directory tree & recreates every symlink (some of which are NTFS junction points), such that there are no junction points on disk when docker tries to commit the RUN step's changes. It works for me locally.

Sadly, I'm getting the same hcsshim error trying to install MSVC build tools on Server Core (but NOT on regular server) images. I checked, and there are no stray junction points created.

@chvjak
Copy link

chvjak commented Feb 25, 2021

with regards cygwin - exatcly my findings. My approch was before sending cygwin to docker - install it on host, and zip instalation folder (to avoid hardlinks), send zip to docker context and unzip in the container.

Sadly, I'm getting the same hcsshim error trying to install MSVC build tools on Server Core (but NOT on regular server) images. I checked, and there are no stray junction points created.

this dockerfile works for me to install build tools on server core

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2019

ADD https://aka.ms/vs/16/release/channel C:\TEMP\VisualStudio.chman
ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe

SHELL ["cmd", "/S", "/C"]


RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache `
    --installPath C:\BuildTools `
    --channelUri C:\Temp\VisualStudio.chman `
    --installChannelUri C:\Temp\VisualStudio.chman `
    --add Microsoft.VisualStudio.Workload.VCTools;includeRecommended `
    --add Microsoft.Component.MSBuild `
    --add Microsoft.VisualStudio.Component.VC.ATL `
    --add Microsoft.VisualStudio.Component.VC.ATLMFC `
|| IF "%ERRORLEVEL%"=="3010" EXIT 0

@lbergnehr
Copy link

I've built a cygwin image successfully with the above script in combination with, before installing cygwin, setting the CYGWIN=winsymlinks env var.

@mloskot mloskot mentioned this issue Jun 18, 2021
Closed
@soroush
Copy link

soroush commented Jul 15, 2021

I am having exact same issue on Windows Server 20H2. None of the proposed workarounds work for me. This happens when trying to commit a fairly large layer (compile qt with vcpkg) at build time. I've also changed the default container size to 500GB.

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  cluster: Manage Mirantis Container Cloud clusters (Mirantis Inc., v1.9.0)
  registry: Manage Docker registries (Docker Inc., 0.1.0)

Server:
 Server Version: 20.10.6
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: none
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 19042 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows Server Datacenter Version 2009 (OS Build 19042.508)
 OSType: windows
 Architecture: x86_64
 CPUs: 48
 Total Memory: 62GiB

@TBBle TBBle mentioned this issue Sep 1, 2021
Open
@Yachtie2020
Copy link

Yachtie2020 commented Sep 24, 2021 •
edited
Loading

I've encountered this issue a number of times recently, like others installing VS Buildtools or any other large program seems to cause it. I was doing a lot of builds, that where failing as I was getting the Dockerfile right and also ctrl+c to kill. This resulted in a number of images (and sometimes containers) labelled NONE (hanging). As per Stack Exchange comments, pruning the containers and images so only my base Windows image remained allowed me to re try and build successfully. I did have the issue a couple times at the end of script creating and I would go back a couple commands in the Dockerfile, add a RUN echo hi to force new cache layers etc, stop at the issue (buildtools) go and remove the RUN echo hi and let it run again. This seem to work too (but I also had far less hanging images from pruning), so at least in my case, I would be suspicious that the cache and/or layers are corrupt or something since clean builds after pruning or forcing some layers to be rebuilt fixed the issue.

FYI I also changed storage-opts to 120GB and ran the build with -m 4GB, but the issue still occurred.

@meirgbinahai
Copy link

For me, I had two commands

COPY settings.default settings
RUN something
COPY . .

I changed it to

COPY . .
RUN something

It's stupid, doesn't make sense, and I have no idea why it happens. All I can say is, it doesn't happen when I copy all the build-context files at once.

@wa6smn
Copy link

wa6smn commented Oct 15, 2021

I solved this problem by allocating a 64 GB disk for my VMware Workstation Pro and initializing it as drive F:. In the DockerEngine settings, I provided Docker with a place to do its work with the data-root entry.
{ "registry-mirrors": [], "insecure-registries": [], "debug": false, "experimental": false, "data-root": "F:\\docker-root" }

@darkvertex
Copy link

darkvertex commented Nov 11, 2021 •
edited
Loading

I got to a point where I had an image that kept dying at build time after a few steps even though I had just about enough free space as the last final image size from the previous build, but it seems a lot of extra space is necessary while building and while running.

Neither hcsshim nor Docker Desktop for Windows gave any comprehensive errors that space was an issue. 🤷‍♂️

As pointed above, I too solved this error by pointing "data-root" to a bigger drive and rebuilding my images.

@JayaRamaRajuGitHub
Copy link

I am still facing the issue. I have added the below Daemon.json file. How to troubleshoot it further?
DockerVersion

{
"storage-opts":["size=200G"],
"data-root": "D:\docker-root"
}
DockerImage

@maranov
Copy link

maranov commented Dec 2, 2022 •
edited
Loading

I've encountered this with Engine 20.10.9 on Windows Server 2019 while building Boost libraries with VS Build Tools and Windows PowerShell. The data root is set to a directory on a secondary drive that's dedicated to Docker and has plenty of free space. Increasing the storage-opts, pruning or a complete reinstall (including removal of files under Program Files, ProgramData and the data root) didn't help. Both the dockerd process and data root are excluded from Defender scans.

What helped, was adding a wait at the end of the RUN script:

sleep -Seconds 300

Why does that work, I have no idea. There shouldn't be anything interfering with the filesystem.

Paradoxically the VS Build Tools and Chocolatey + CMake installation in one of the previous steps is not a problem.

@gabyx
Copy link

gabyx commented Apr 12, 2023

@maranov: Facing exeactly the same problem. Did you solve this differently???

@wa6smn
Copy link

wa6smn commented Apr 12, 2023 via email

@mpashaasu
Copy link

mpashaasu commented Dec 22, 2023 •
edited
Loading

Run in CMD/PS as admin

docker build -t test:v1 . -m 4GB

with below configs

{
"experimental": false,
"storage-opts": [
"size=150GB"
],
"data-root": "C:\\docker-root"
}

base image is
mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2022

during intermediate run I think the size crosses 20GB

Gets below error -

re-exec error: exit status 1: output: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)

Very annoying is there a fix for this?

@wa6smn
Copy link

wa6smn commented Dec 22, 2023 via email

@mpashaasu
Copy link

data-root has around 450GB of space on the disk

@wa6smn
Copy link

wa6smn commented Dec 22, 2023 via email

@mpashaasu
Copy link

mpashaasu commented Dec 26, 2023 •
edited
Loading

\\ is already present in my config, the parser will anyway warn you in the docker desktop UI.

@doctorpangloss
Copy link

Does COPY --link interact with this issue?

@4ampro
Copy link

4ampro commented Mar 21, 2024 •
edited
Loading

I'm having a similar issue with the PrepareLayer but it all started with the msg box, "Docker Service is not running". Using the data-root parameter, I pointed my config.json to a drive with space however, the file not found seems to be "c:\bcartifacts.cache:c:\dl".

The system cannot find the path specified. (0x3).
ExitCode: 125

FORGOT TO ADD THIS:
--volume "c:\bcartifacts.cache:c:\dl"

Please kindly let me know if I should post separately?
Any help is appreciated.

More infos available if needed.

win 10 pro
docker 4.28.0 (139021)
pse as admin
new-bccontainer
bc dev license v22

Edit: gave it a proper issue of its own:
#2075

@rafabios
Copy link

I had the same issue, for me it was due to memory/cpu usage ( I was running an virtual machine as well). When I drop the vm, it builded good.

@arkadR
Copy link

arkadR commented Jun 11, 2024 •
edited
Loading

We're still randomly facing this issue when building images based servercore:ltsc2022 images on EC2 build agents. None of the workarounds listed in this thread seem to have helped. What we tried:

  • Increasing VM CPU and memory
  • Storage-opts up to 150GB
  • Sleep at the end finishing the problematic build step
  • docker system prune and other system cleanups

This error does pop up randomly (about 30% of times) and we really can't figure out what's causing it.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests