Docker keeps taking more storage space under C:\ProgramData\docker\windowsfilter with no way to reclaim it since prune commands not available

[paylocity-sflanders]

Expected behavior

Drive space grows and shrinks as you add and remove things and cleanup orphaned data.

Actual behavior

With no images or containers installed we are seeing 30+ gigs of files under C:\ProgramData\docker\windowsfilter. It appears to be caching things that i explicitly removed.

Information

We have been working on some docker images where we are constantly running the build command. We have a helper script that handles cleaning up unused data but it only cleans up part of it. The C:\ProgramData\docker\windowsfilter folder just keeps growing and we have to continually fight running out of drive space. The only solution i'm seeing now is to completely uninstall docker and reinstall, but thats not an option if we use this in production. We need reliable way to manage storage usage in docker for it to be viable.

Windows Server 2016 DataCenter, Version 1607, OS Build 14393.447

Is there any workarounds to cleanup unused docker artifacts? The prune command seem to help with this but they don't appear to be available on windows server 2016, only windows 10.

docker system prune
docker container prune
docker image prune
docker network prune
docker volume prune

Steps to reproduce the behavior

Simply interacting with docker by building and running images seems to continually add data to C:\ProgramData\docker\windowsfilter. Below is more information on what we are seeing in our environment

Images Before

PS C:\Users\SFlanders\Desktop\images\msbuild> docker images -a
REPOSITORY                                           TAG                    IMAGE ID            CREATED             SIZE
microsoft/dotnet-samples                             dotnetapp-nanoserver   12eecaee618f        11 days ago         1.11 GB
docker.paylocity.com:5000/microsoft/dotnet-samples   dotnetapp-nanoserver   12eecaee618f        11 days ago         1.11 GB
microsoft/windowsservercore                          10.0.14393.953         b4713e4d8bab        7 weeks ago         10.1 GB
docker.paylocity.com:5000/microsoft/nanoserver       10.0.14393.953         18a0d32a4b98        7 weeks ago         1 GB
microsoft/nanoserver                                 10.0.14393.953         18a0d32a4b98        7 weeks ago         1 GB

Containers Before

PS C:\Users\SFlanders\Desktop\images\msbuild> docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Volumes Before

PS C:\Users\SFlanders\Desktop\images\msbuild> docker volume ls
DRIVER              NAME

Docker Folders Before

Path                                  TotalGBytes DirCount FileCount TimeElapsed
----                                  ----------- -------- --------- -----------
C:\ProgramData\docker\config               0.0000        1         1      0.0190
C:\ProgramData\docker\containers                0        1         0      0.0130
C:\ProgramData\docker\credentialspecs           0        1         0      0.0140
C:\ProgramData\docker\image                0.0281       87       382      0.0980
C:\ProgramData\docker\network              0.0001        2         1      0.0250
C:\ProgramData\docker\swarm                     0        1         0      0.0130
C:\ProgramData\docker\tmp                       0        1         0      0.0140
C:\ProgramData\docker\trust                     0        1         0      0.0140
C:\ProgramData\docker\volumes              0.0000        1         2      0.0150
C:\ProgramData\docker\windowsfilter       35.4629    69206    258033     45.8145

Then we deleted all the images and expected to see a great reduction in storage, but it was only reduced partially

PS C:\Users\SFlanders\Desktop\images> docker images -q  | % { docker rmi $_ -f }
PS C:\Users\SFlanders\Desktop\images> docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

Docker Folders After

Path                                  TotalGBytes DirCount FileCount TimeElapsed
----                                  ----------- -------- --------- -----------
C:\ProgramData\docker\config               0.0000        1         1      0.0270
C:\ProgramData\docker\containers                0        1         0      0.0160
C:\ProgramData\docker\credentialspecs           0        1         0      0.0140
C:\ProgramData\docker\image                0.0243       76       322      0.0900
C:\ProgramData\docker\network              0.0001        2         1      0.0210
C:\ProgramData\docker\swarm                     0        1         0      0.0170
C:\ProgramData\docker\tmp                       0        1         0      0.0160
C:\ProgramData\docker\trust                     0        1         0      0.0140
C:\ProgramData\docker\volumes              0.0000        1         2      0.0150
C:\ProgramData\docker\windowsfilter       31.9801    62041    218368     38.5320

Pulling images i just removed are still on the box

PS C:\Users\SFlanders\Desktop\images> docker pull microsoft/windowsservercore:10.0.14393.953
10.0.14393.953: Pulling from microsoft/windowsservercore
3889bb8d808b: Already exists
503d87f3196a: Already exists
Digest: sha256:0565e89ad5e54bada152062976c80524c3a46ec52af2a54bc29f8e9d1d9e8f1b
Status: Downloaded newer image for microsoft/windowsservercore:10.0.14393.953

Docker Info

PS C:\Users\SFlanders\Desktop\images> docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.2-cs2-ws-beta
Storage Driver: windowsfilter
 Windows: 
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: nat null overlay transparent
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.447.amd64fre.rs1_release_inmarket.161102-0100)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 1
Total Memory: 4 GiB
Name: QA-Sandbox-77
ID: YEEJ:MRKB:2LIJ:N7DN:EU2I:3DND:SI6S:E24M:IF34:6Q2W:R6WL:OBB7
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Docker version

SPS C:\Users\SFlanders\Desktop\images> docker version
Client:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Server:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35

Output from Debug-ContainerHost.ps1

PS C:\Users\SFlanders\Desktop\images> .\Debug-ContainerHost.ps1
Checking for common problems
Describing Windows Version and Prerequisites
 [+] Is Windows 10 Anniversary Update or Windows Server 2016 786ms
 [+] Has KB3192366, KB3194496, or later installed if running Windows build 14393 196ms
 [+] Is not a build with blocking issues 34ms
 [+] Has 'Containers' feature installed 5.34s
Describing Docker is installed
 [+] A Docker service is installed - 'Docker' or 'com.Docker.Service'  148ms
 [+] Service is running 35ms
 [+] Docker.exe is in path 2.17s
 [-] Docker is registered in the EventLog service 194ms
   Expected: {True}
   But was:  {False}
   71:         (Test-Path "HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application\docker") -or (Test-Path "HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application\DockerService") |
 Should Be $true
   at <ScriptBlock>, C:\Users\SFlanders\Desktop\images\Debug-ContainerHost.ps1: line 71
Describing User has permissions to use Docker daemon
 [+] docker.exe should not return access denied 80ms
Describing Windows container settings are correct
 [+] Do not have DisableVSmbOplock set to 1 69ms
 [+] Do not have zz values set 71ms
 [+] Do not have FDVDenyWriteAccess set to 1 20ms
Describing The right container base images are installed
 [-] At least one of 'microsoft/windowsservercore' or 'microsoft/nanoserver' should be installed 151ms
   ValidationMetadataException: The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
   ParameterBindingValidationException: Cannot validate argument on parameter 'Property'. The argument is null or empty. Provide an argument that is not null or empty, and then try the comman
d again.
   at <ScriptBlock>, C:\Users\SFlanders\Desktop\images\Debug-ContainerHost.ps1: line 115
Describing Container network is created
 [+] At least one local container network is available 3.79s
 [+] At least one NAT, Transparent, or L2Bridge Network exists 52ms
 [+] NAT Network's vSwitch is internal 207ms
 [+] A Windows NAT is configured if a Docker NAT network exists 416ms
 [+] Specified Network Gateway IP for NAT network is assigned to Host vNIC 123ms
 [-] NAT Network's internal prefix does not overlap with external IP' 123ms
   Expected {0} to be greater than {0}
   252:         $hostips.Count | Should BeGreaterThan 0
   at <ScriptBlock>, C:\Users\SFlanders\Desktop\images\Debug-ContainerHost.ps1: line 252
Showing output from: docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.2-cs2-ws-beta
Storage Driver: windowsfilter
 Windows: 
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: nat null overlay transparent
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.447.amd64fre.rs1_release_inmarket.161102-0100)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 1
Total Memory: 4 GiB
Name: QA-Sandbox-77
ID: YEEJ:MRKB:2LIJ:N7DN:EU2I:3DND:SI6S:E24M:IF34:6Q2W:R6WL:OBB7
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Showing output from: docker version
Client:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Server:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Showing output from: docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
fd8b16fc1315        LocalNetwork        transparent         local
6dd384f287c7        nat                 nat                 local
e784399a7699        none                null                local

Getting Warnings & errors in the Windows event logs from the last 24 hours
Logs saved to C:\Users\SFlanders\Desktop\images\logs_20170430-115016.csv


Getting Docker for Windows daemon logs from the last execution
    Note: More logs are available at C:\Users\SFlanders\AppData\Local\Docker. Only showing the latest.
Get-Content : Cannot find path 'C:\Users\SFlanders\AppData\Local\Docker\log.txt' because it does not exist.
At C:\Users\SFlanders\Desktop\images\Debug-ContainerHost.ps1:291 char:1
+ Get-Content "$($ENV:LOCALAPPDATA)\Docker\log.txt" | Select-String "Wi ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\SFland...\Docker\log.txt:String) [Get-Content], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand

Cleanup script

docker ps -a -q | % { docker stop $_ }
docker ps -a -q | % { docker rm $_ }
docker images --filter "dangling=true" -q --no-trunc | % { docker rmi $_ -f }
docker volume ls -qf dangling=true | % { docker volume rm $_ }

[friism]

@paylocity-sflanders you're using an old version of Docker that doesn't support prune. The latest stable version is 1.13.1 so I recommend you upgrade at least to that (for testing advanced stuff you might want to look at un-supported 17.04 or 17.05-rcs: https://github.com/moby/moby/releases - but don't use 17.05.0-rc2, it has bugs). Here's how to upgrade to latest stable:

Install-Package -Name docker -ProviderName DockerMsftProvider -Verbose -Update

If all else fails, have a look at this: https://github.com/jhowardmsft/docker-ci-zap But know that it's very dangerous and very unsupported.

[friism]

Closing, but please re-open if this doesn't get you unstuck.

[paylocity-sflanders]

I'm noticing different behavior if running the same version on windows server 2016 and windows 10. On server it doesn't appear to cleanup the same amount of stuff. In this example i run the same command to delete the image, prune and then pull again. You will see on windows 10 it had to download several more layers than on server 2016, so it has the appearance that some of the layers don't get removed.

docker version
docker image rm docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0 -f
docker image prune -f
docker pull docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0

Windows Server 2016 Standard (1607, 14393.447)

PS C:\Users\SFlanders\Desktop\Build.Scripts> docker version
Client:
 Version:      17.03.1-ee-3
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   3fcee33
 Built:        Thu Mar 30 19:31:22 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.03.1-ee-3
 API version:  1.27 (minimum version 1.24)
 Go version:   go1.7.5
 Git commit:   3fcee33
 Built:        Thu Mar 30 19:31:22 2017
 OS/Arch:      windows/amd64
 Experimental: false
PS C:\Users\SFlanders\Desktop\Build.Scripts> docker image rm docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0 -f
Untagged: docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0
Untagged: docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build@sha256:6d93ec013ebb02315b71ee866fcde4384ac1f0c05398e0634e1065dac5a90043
Deleted: sha256:505fccd3d7e5085bca6768dacc3f594554776cdc629fb2626be80c9a6cc00ea9
Deleted: sha256:51306f576c3b94b1ac3545056fa300079971b8e4bddaa9bcf5de66140b92b8dd
Deleted: sha256:f4fa48e813b5a6cd4bf6c1cfb71191ae3f34fe4e7a54f144a53bd2a75b9253bf
Deleted: sha256:9dd552f8aebde76055176b791d6ed66847ff4f5aca7d2e48612b8a345f0a7137
Deleted: sha256:18cfb346bb12168e46e9b58e9fb6486dfae3cde6794ebd887b2442be55cd76fa
Deleted: sha256:3cbd42c797c4a39ad73c65a25ca412cb8a98a33041028996cf81c9728721f142
Deleted: sha256:bb986a3e598754e2ceb29135b81d31041d095a93293a7fcdf073ff440279b2c8
Deleted: sha256:6e659a5894c5e9e84fc5bb62b31b7b6341132340b8f708add231d4ce99b7a4fa
PS C:\Users\SFlanders\Desktop\Build.Scripts> docker image prune -f
Total reclaimed space: 0 B
PS C:\Users\SFlanders\Desktop\Build.Scripts> docker pull docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0
1.0.0: Pulling from paylocity/wsc-dotnet-v15-452-build
3889bb8d808b: Already exists
503d87f3196a: Already exists
876c6fd7689c: Already exists
deda1fa6037c: Already exists
44666fec49b4: Already exists
b07c33049823: Already exists
ca31f682a768: Already exists
b46addb5ed15: Already exists
49ef4ef33eaa: Already exists
b28e6bcbaa6f: Already exists
d3ea633a6176: Already exists
4d0a6d0cff19: Already exists
eac561e587a4: Already exists
b6d79ea2d0cf: Already exists
31687b1c3e95: Already exists
bdaabbbddac7: Already exists
81df1ab623bf: Already exists
7b8f4d845ba3: Already exists
a24f8908105e: Already exists
b1b51a829c3b: Pull complete
146472b463e5: Pull complete
6acb0f73bcb3: Pull complete
91924a0c210d: Pull complete
51bed710962a: Pull complete
8d1dffa3f6d7: Pull complete
4ac3fddb76b4: Pull complete
Digest: sha256:6d93ec013ebb02315b71ee866fcde4384ac1f0c05398e0634e1065dac5a90043
Status: Downloaded newer image for docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0
PS C:\Users\SFlanders\Desktop\Build.Scripts>

Windows 10 Enterprise (1607, 14393.1066)

C:\Windows\system32>docker version
Client:
 Version:      17.03.1-ee-3
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   3fcee33
 Built:        Thu Mar 30 19:31:22 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.03.1-ee-3
 API version:  1.27 (minimum version 1.24)
 Go version:   go1.7.5
 Git commit:   3fcee33
 Built:        Thu Mar 30 19:31:22 2017
 OS/Arch:      windows/amd64
 Experimental: false

C:\Windows\system32>docker image rm docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0 -f
Error response from daemon: No such image: docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0

C:\Windows\system32>docker image prune -f
Total reclaimed space: 0 B

C:\Windows\system32>docker pull docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0
1.0.0: Pulling from paylocity/wsc-dotnet-v15-452-build
3889bb8d808b: Already exists
503d87f3196a: Already exists
876c6fd7689c: Pull complete
deda1fa6037c: Pull complete
44666fec49b4: Pull complete
b07c33049823: Pull complete
ca31f682a768: Pull complete
b46addb5ed15: Pull complete
49ef4ef33eaa: Pull complete
b28e6bcbaa6f: Pull complete
d3ea633a6176: Pull complete
4d0a6d0cff19: Pull complete
eac561e587a4: Pull complete
b6d79ea2d0cf: Pull complete
31687b1c3e95: Pull complete
bdaabbbddac7: Pull complete
81df1ab623bf: Pull complete
7b8f4d845ba3: Pull complete
a24f8908105e: Pull complete
b1b51a829c3b: Pull complete
146472b463e5: Pull complete
6acb0f73bcb3: Pull complete
91924a0c210d: Pull complete
51bed710962a: Pull complete
8d1dffa3f6d7: Pull complete
4ac3fddb76b4: Pull complete
Digest: sha256:6d93ec013ebb02315b71ee866fcde4384ac1f0c05398e0634e1065dac5a90043
Status: Downloaded newer image for docker.paylocity.com:5000/paylocity/wsc-dotnet-v15-452-build:1.0.0

[paylocity-sflanders]

I can't reopen this, so created new ticket and linked to this one. The new one is: #745

[frankfuu]

hi @paylocity-sflanders ,

Can I ask what script you have to get these results?

Path                                  TotalGBytes DirCount FileCount TimeElapsed
----                                  ----------- -------- --------- -----------
C:\ProgramData\docker\config               0.0000        1         1      0.0190
C:\ProgramData\docker\containers                0        1         0      0.0130
C:\ProgramData\docker\credentialspecs           0        1         0      0.0140
C:\ProgramData\docker\image                0.0281       87       382      0.0980
C:\ProgramData\docker\network              0.0001        2         1      0.0250
C:\ProgramData\docker\swarm                     0        1         0      0.0130
C:\ProgramData\docker\tmp                       0        1         0      0.0140
C:\ProgramData\docker\trust                     0        1         0      0.0140
C:\ProgramData\docker\volumes              0.0000        1         2      0.0150
C:\ProgramData\docker\windowsfilter       35.4629    69206    258033     45.8145

[paylocity-sflanders]

https://www.powershelladmin.com/wiki/Get_Folder_Size_with_PowerShell,_Blazingly_Fast

[gparlakov]

I've experienced the same behavior (images remaining on disk even after docker image remove) on Windows 10 Pro.
And using Docker - Settings - Reset - Reset to factory defaults did the trick for me.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked