'docker push' w/ docker-registry on reencrypt route with SSL certs fails with "unauthorized: authentication required"

[dmsimard]

In summary, a docker-registry service on a reencrypt route (with proper certificates) and self-signed certificates mounted in /etc/secrets will let you login successfully but pushes will fail with a client-side error: unauthorized: authentication required

Switching the route to passthrough and re-mounting /etc/secrets with the proper certificates lets you both log in and push.

Reproducible every time when deploying origin 1.5 with openshift-ansible on CentOS mostly following instructions provided for standalone registry setup.

Version

oc v1.5.0
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.1.6:8443
openshift v1.5.0
kubernetes v1.5.2+43a9be4

Steps To Reproduce

1. Create proper certificates

2. Configure openshift-ansible to set up route host, termination and ssl certs, for example:

openshift_hosted_registry_routehost: registry.domain.tld
openshift_hosted_registry_routetermination: reencrypt
openshift_hosted_registry_routecertificates:
  certfile: "/etc/letsencrypt/live/registry.domain.tld/registry.domain.tld-cert.pem"
  keyfile: "/etc/letsencrypt/live/registry.domain.tld/registry.domain.tld-privkey.pem"
  cafile: "/etc/letsencrypt/live/registry.domain.tld/registry.domain.tld-chain.pem"

3. Run openshift-ansible to set up a standalone registry+console as per documentation

4. Create user and project

5. Retrieve user token and login successfully

docker login -p token -e unused -u unused registry.domain.tld

6. Pull an image from docker hub and try to push it to the project

docker pull alpine
docker tag docker.io/alpine registry.domain.tld/project/alpine
docker push registry.domain.tld/project/alpine

Push fails with:

# docker push registry.domain.tld/test/alpine
The push refers to a repository [registry.domain.tld/test/alpine]
e154057080f4: Pushing [==================================================>] 4.016 MB
unauthorized: authentication required

Current Result

Login works but push fails

Expected Result

Login and push both work

Additional Information

Here's a side-by-side diff of route/svc/pod export as well as master-config.yml for the exact same deployment configuration except one is passthrough (left), the other is reencrypt (right):
https://www.diffchecker.com/wzWtOFmJ

Here's the docker-registry logs on a successful docker login:

10.128.0.1 - - [18/May/2017:17:00:53 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:00:53.526338597Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=8556f42f-bf65-413e-9d76-1faafbd51ca9 http.request.method=GET http.request.remoteaddr="10.128.0.1:37868" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:00:53.52642273Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=8556f42f-bf65-413e-9d76-1faafbd51ca9 http.request.method=GET http.request.remoteaddr="10.128.0.1:37868" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:00:53.954144006Z" level=info msg="response completed" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=e7e72bda-6c08-4212-93f2-e72ae7e2d624 http.request.method=GET http.request.remoteaddr="10.128.0.1:37872" http.request.uri="/openshift/token?account=unused&client_id=docker&offline_token=true" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.contenttype="application/json" http.response.duration=3.257955ms http.response.status=200 http.response.written=117 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:00:53 +0000] "GET /openshift/token?account=unused&client_id=docker&offline_token=true HTTP/1.1" 200 117 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:00:54.336259375Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=ed5212a5-f786-4b4d-bdac-93221798a762 http.request.method=GET http.request.remoteaddr="10.128.0.1:37876" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:00:54.339158083Z" level=info msg="response completed" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=ed5212a5-f786-4b4d-bdac-93221798a762 http.request.method=GET http.request.remoteaddr="10.128.0.1:37876" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.233502ms http.response.status=200 http.response.written=2 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:00:54 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"

Here's the docker-registry logs on a failed docker push:

time="2017-05-18T17:01:01.12733397Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=b363194a-7507-4891-8556-ad401ff5c106 http.request.method=GET http.request.remoteaddr="10.128.0.1:37900" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:01:01.127436592Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=b363194a-7507-4891-8556-ad401ff5c106 http.request.method=GET http.request.remoteaddr="10.128.0.1:37900" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:01:01 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:01:01.509111676Z" level=info msg="response completed" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=2e668396-a92b-484c-8ca9-405b93d8d2b7 http.request.method=GET http.request.remoteaddr="10.128.0.1:37904" http.request.uri="/openshift/token?account=unused&scope=repository%3Atripleo%2Fcentos%3Apush%2Cpull" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.contenttype="application/json" http.response.duration=3.634646ms http.response.status=200 http.response.written=117 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:01:01 +0000] "GET /openshift/token?account=unused&scope=repository%3Atripleo%2Fcentos%3Apush%2Cpull HTTP/1.1" 200 117 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:01:02.01261792Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.01812253Z" level=debug msg="Origin auth: checking for access to repository:tripleo/centos:pull" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.020825074Z" level=debug msg="Origin auth: checking for access to repository:tripleo/centos:push" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.023760248Z" level=debug msg="(*linkedBlobStore).Writer" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.030031724Z" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/tripleo/centos/_uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8/startedat\")" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry trace.duration=6.198524ms trace.file="/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=abbf9ec5-71c8-4186-9c67-05db49d2ed50 trace.line=95 vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.030201053Z" level=debug msg="filesystem.Writer(\"/docker/registry/v2/repositories/tripleo/centos/_uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8/data\", false)" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry trace.duration=86.493µs trace.file="/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=cc01145a-bc40-40ff-841e-0e9821252df5 trace.line=124 vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.035808284Z" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/tripleo/centos/_uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8/hashstates/sha256/0\")" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry trace.duration=5.454152ms trace.file="/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=a98da2cc-1181-4afa-aa41-1c9e8a711217 trace.line=95 vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.036032606Z" level=info msg="response completed" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.duration=27.502021ms http.response.status=202 http.response.written=0 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:01:02 +0000] "POST /v2/tripleo/centos/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:01:02.408410116Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=54b68a50-9ff4-45fb-b3dc-9b4667694cde http.request.method=PATCH http.request.remoteaddr="10.128.0.1:37912" http.request.uri="/v2/tripleo/centos/blobs/uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8?_state=fapf89621-4skE0-D_L9JrmM1gpUNTB3_hhcq9BG8OR7Ik5hbWUiOiJ0cmlwbGVvL2NlbnRvcyIsIlVVSUQiOiIzN2ZlNGVlMC0yMjQ1LTRjODktOGY5ZS0zMGM3NGYxNmIxZTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMThUMTc6MDE6MDIuMDIzNzk4MjEzWiJ9" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry vars.name="tripleo/centos" vars.uuid=37fe4ee0-2245-4c89-8f9e-30c74f16b1e8 
time="2017-05-18T17:01:02.408521996Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=54b68a50-9ff4-45fb-b3dc-9b4667694cde http.request.method=PATCH http.request.remoteaddr="10.128.0.1:37912" http.request.uri="/v2/tripleo/centos/blobs/uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8?_state=fapf89621-4skE0-D_L9JrmM1gpUNTB3_hhcq9BG8OR7Ik5hbWUiOiJ0cmlwbGVvL2NlbnRvcyIsIlVVSUQiOiIzN2ZlNGVlMC0yMjQ1LTRjODktOGY5ZS0zMGM3NGYxNmIxZTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMThUMTc6MDE6MDIuMDIzNzk4MjEzWiJ9" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry vars.name="tripleo/centos" vars.uuid=37fe4ee0-2245-4c89-8f9e-30c74f16b1e8 
10.128.0.1 - - [18/May/2017:17:01:02 +0000] "PATCH /v2/tripleo/centos/blobs/uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8?_state=fapf89621-4skE0-D_L9JrmM1gpUNTB3_hhcq9BG8OR7Ik5hbWUiOiJ0cmlwbGVvL2NlbnRvcyIsIlVVSUQiOiIzN2ZlNGVlMC0yMjQ1LTRjODktOGY5ZS0zMGM3NGYxNmIxZTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMThUMTc6MDE6MDIuMDIzNzk4MjEzWiJ9 HTTP/1.1" 401 208 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"

[dmsimard]

@abutcher @aweiteka fyi, as per discussed

[mjudeikis]

Does this has a workaround now?

[mjudeikis]

Bizarre thing is that some of the layers managed to land in the filesystem:

drwxr-sr-x. 17 1000000000 1000000000 4096 Aug 18 12:36 .                                                                                                                                                                                             
drwxr-sr-x.  3 1000000000 1000000000   22 Aug 18 12:23 ..                                                                                                                                                                                            
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:27 13e33fe4-12eb-4924-9bbf-cb72a79a6caa                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:31 150af01b-7c2a-44b9-9c68-a30bc15a8933                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:31 29aa3a9e-dc4a-4d91-8fce-4be78b094015                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:31 450ec39a-a94c-4b23-beac-cd68892f360e                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:33 50ef1402-e1b1-4c68-af34-b34db4bce3df                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:23 7037d840-c5cf-4716-b98c-17cea1d862e3                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:27 7f57b5ef-fb9b-4e2d-8a76-0c15b157224d                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:36 882b3a66-45d9-41d7-83b2-abceb7fc21cb                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:33 9be1f361-d503-4653-ac42-41d42b9c21a4                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:23 c20ae65b-b249-4e0e-a95e-3fae9a9f494f                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:33 d16cd403-c808-4e14-b035-ea3f89a23597                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:23 d20250d6-d738-4f70-bc94-0f29bfe6a030                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:36 dcc1b319-0ecf-4c14-a344-f047fea6fb9e                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:27 e5eb03da-d316-4439-a0f7-1d18f38005a9                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:36 f8caac91-12bf-4adc-af2f-d54691968457                                                                                                                                                          
sh-4.2$ pwd                                                                                                                                                                                                                                          
/registry/docker/registry/v2/repositories/library/rhel-tools/_uploads 

and docker cli:

[root@ocpt00236 ~]# docker push  containers.registry.company.local/library/rhel-tools:latest
The push refers to a repository [containers.registry.company.local/library/rhel-tools]
d466c001c314: Pushing [==================================================>]  1.19 GB
bb4f52dd78f6: Pushing [==================================================>] 10.24 kB
f1f88d1c363a: Pushing [==================================================>] 195.9 MB/195.9 MB
unauthorized: authentication required

Logs:

10.217.4.1 - - [18/Aug/2017:12:36:49 +0000] "PATCH /v2/library/rhel-tools/blobs/uploads/882b3a66-45d9-41d7-83b2-abceb7fc21cb?_state=mvg6sxRQPHJ6kjsn_He6J6ksLpsFMx1NR2iNjnSb2wN7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiODgyYjNhNjYtNDVkOS00MWQ3LTgzYjItYWJjZWI3ZmMyMWNiIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyODY0NTY4OVoifQ%3D%3D HTTP/1.1" 401 216 "" "docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"
--
  | time="2017-08-18T12:36:50.127546721Z" level=debug msg="authorizing request" go.version=go1.7.6 http.request.host="containers.registry.companys.local:443" http.request.id=cc250e82-ea11-4e2d-bc01-a167e43ba89f http.request.method=PATCH http.request.remoteaddr="10.217.2.1:46762" http.request.uri="/v2/library/rhel-tools/blobs/uploads/f8caac91-12bf-4adc-af2f-d54691968457?_state=9nBbILDnjl1MwyPGVu1sgaXC6hJeh9KFkvg0gAqD81Z7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZjhjYWFjOTEtMTJiZi00YWRjLWFmMmYtZDU0NjkxOTY4NDU3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyNzQ4MjA2OFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=f8caac91-12bf-4adc-af2f-d54691968457
  | time="2017-08-18T12:36:50.12770762Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.6 http.request.host="containers.registry.comapny.local:443" http.request.id=cc250e82-ea11-4e2d-bc01-a167e43ba89f http.request.method=PATCH http.request.remoteaddr="10.217.2.1:46762" http.request.uri="/v2/library/rhel-tools/blobs/uploads/f8caac91-12bf-4adc-af2f-d54691968457?_state=9nBbILDnjl1MwyPGVu1sgaXC6hJeh9KFkvg0gAqD81Z7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZjhjYWFjOTEtMTJiZi00YWRjLWFmMmYtZDU0NjkxOTY4NDU3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyNzQ4MjA2OFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=f8caac91-12bf-4adc-af2f-d54691968457
  | time="2017-08-18T12:36:50.132647441Z" level=debug msg="authorizing request" go.version=go1.7.6 http.request.host="containers.registry.company.local:443" http.request.id=8765f19b-d178-4166-97b4-e123174f7ae2 http.request.method=PATCH http.request.remoteaddr="10.217.0.1:52708" http.request.uri="/v2/library/rhel-tools/blobs/uploads/dcc1b319-0ecf-4c14-a344-f047fea6fb9e?_state=t24iCUNB4RE4IT740ZVaG4pkelQzRU_jCcPk5H1t5wh7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZGNjMWIzMTktMGVjZi00YzE0LWEzNDQtZjA0N2ZlYTZmYjllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyMjE1NjkyOFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=dcc1b319-0ecf-4c14-a344-f047fea6fb9e
  | time="2017-08-18T12:36:50.132767552Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.6 http.request.host="containers.registry.company.local:443" http.request.id=8765f19b-d178-4166-97b4-e123174f7ae2 http.request.method=PATCH http.request.remoteaddr="10.217.0.1:52708" http.request.uri="/v2/library/rhel-tools/blobs/uploads/dcc1b319-0ecf-4c14-a344-f047fea6fb9e?_state=t24iCUNB4RE4IT740ZVaG4pkelQzRU_jCcPk5H1t5wh7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZGNjMWIzMTktMGVjZi00YzE0LWEzNDQtZjA0N2ZlYTZmYjllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyMjE1NjkyOFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=dcc1b319-0ecf-4c14-a344-f047fea6fb9e
  | 10.217.0.1 - - [18/Aug/2017:12:36:50 +0000] "PATCH /v2/library/rhel-tools/blobs/uploads/dcc1b319-0ecf-4c14-a344-f047fea6fb9e?_state=t24iCUNB4RE4IT740ZVaG4pkelQzRU_jCcPk5H1t5wh7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZGNjMWIzMTktMGVjZi00YzE0LWEzNDQtZjA0N2ZlYTZmYjllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyMjE1NjkyOFoifQ%3D%3D HTTP/1.1" 401 216 "" "docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"
  | 10.217.2.1 - - [18/Aug/2017:12:36:50 +0000] "PATCH /v2/library/rhel-tools/blobs/uploads/f8caac91-12bf-4adc-af2f-d54691968457?_state=9nBbILDnjl1MwyPGVu1sgaXC6hJeh9KFkvg0gAqD81Z7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZjhjYWFjOTEtMTJiZi00YWRjLWFmMmYtZDU0NjkxOTY4NDU3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyNzQ4MjA2OFoifQ%3D%3D HTTP/1.1" 401 216 "" "docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"

[miminar]

Resolved in 3.7 via #15694.

See bz#1471707 for more details.

If you need it in earlier release, please clone the bugzilla with a desired target release.