Docker Push Retrying Error: Private Registry Fix
Hello everyone, today I’d like to share some other gotcha about a private docker registry solution hidden behind nginx reverse proxy.
This information may be helpful to you if you have direct access to the server and administer private docker registry yourself.
Issue
After system updates once working system suddenly has stopped functioning properly on any attempt of doing docker push.
Pulling an existing image was working perfectly though.
My console output looked like on docker push attempt.
`67ba809ad0e044: Preparing 6831bd36e157b6: Preparing 696185ed1ad590: Preparing 70f5600c6330da: Preparing 7131bd36e157b6: Retrying in 5 seconds 7231bd36e157b6: Retrying in 4 seconds 7331bd36e157b6: Retrying in 3 seconds 7431bd36e157b6: Retrying in 2 seconds 75a4b9c6e7f8bb: Retrying in 5 seconds 7631bd36e157b6: Retrying in 1 second 77a4b9c6e7f8bb: Retrying in 4 seconds`And it was continuously trying to repeat push operation without any success. No matter whether it was from my local machine or from build server – same results.
Solution
I realized the problem may be related to nginx settings for reverse proxy.
Assuming a docker registry accessible from registry.example.com:5000 but hosted internally on the port 5001 the nginx reverse proxy settings may look like this:
`server { listen 5000 ssl; listen [::]:5000 ssl; server_name registry.example.com; client_max_body_size 500M; location / { set $upstream localhost:5001; proxy_pass https://localhost:5001/; proxy_http_version 1.1; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection keep-alive; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Ssl on; proxy_set_header X-Forwarded-Proto https; proxy_read_timeout 900; proxy_request_buffering off; proxy_cache off; proxy_buffering off; }
ssl_certificate /your/path/to/fullchain.pem;
ssl_certificate_key /your/path/to/privkey.pem;
}In most cases the settings are self explaining, but what you may particularly be interested in is the settings to the cache and buffering`.
My general recommendation would be to stay with the original solution as long as it’s possible without putting any reverse proxy between registry server and the client. There are plenty of good to go solutions available out of the box:
- GitLab Container Registry
- Docker Registry Server
- Harbor
Even more are available online as a service.
I hope this information was helpful to you. If you have some good private docker registry solution or you have any question don’t hesitate to drop me a message. You can find my contact channels below.
Take care, Ievgen
Frequently Asked Questions
How do I fix the Docker push 'Retrying in N seconds' error with a private registry behind nginx?
When docker push repeatedly shows 'Retrying in N seconds' against a private registry behind an nginx reverse proxy, the cause is usually nginx buffering and caching settings. Set proxy_request_buffering off, proxy_cache off, and proxy_buffering off in the nginx location block, and raise client_max_body_size so large layers can pass through.
Why can I pull but not push to my private Docker registry?
If docker pull works but docker push fails with retrying errors against a private registry, the reverse proxy in front of the registry is often the problem rather than the registry itself. An nginx reverse proxy with default buffering and caching can break the upload of image layers while still allowing downloads to succeed.
Which nginx settings matter for a Docker registry reverse proxy?
For an nginx reverse proxy in front of a Docker registry, the settings that matter most are proxy_request_buffering off, proxy_buffering off, and proxy_cache off, plus a large client_max_body_size such as 500M to allow big image layers. Forwarding headers like Host, X-Forwarded-Proto, and X-Real-IP should also be set.
What private Docker registry options avoid reverse proxy problems?
To avoid reverse proxy issues entirely, run a ready-made registry rather than placing nginx between the registry and client when possible. Options include the GitLab Container Registry, the Docker Registry Server, and Harbor. Hosted registry services are also available if you prefer not to self-administer one.