Docker Push Error With Private Registry – Retrying In N Seconds

Hello everyone, today I’d like to share some other gotcha about a private docker registry solution hidden behind nginx reverse proxy.

This information may be helpful to you if you have direct access to the server and administer private docker registry yourself.

Issue

After system updates once working system suddenly has stopped functioning properly on any attempt of doing docker push.

Pulling an existing image was working perfectly though.

My console output looked like on docker push attempt.

67ba809ad0e044: Preparing
6831bd36e157b6: Preparing
696185ed1ad590: Preparing
70f5600c6330da: Preparing
7131bd36e157b6: Retrying in 5 seconds
7231bd36e157b6: Retrying in 4 seconds
7331bd36e157b6: Retrying in 3 seconds
7431bd36e157b6: Retrying in 2 seconds
75a4b9c6e7f8bb: Retrying in 5 seconds
7631bd36e157b6: Retrying in 1 second
77a4b9c6e7f8bb: Retrying in 4 seconds

And it was continuously trying to repeat push operation without any success. No matter whether it was from my local machine or from build server – same results.

Solution

I realized the problem may be related to nginx settings for reverse proxy.

Assuming a docker registry accessible from registry.example.com:5000 but hosted internally on the port 5001 the nginx reverse proxy settings may look like this:

server {
    listen 5000 ssl;
    listen [::]:5000 ssl;
    server_name registry.example.com;
    client_max_body_size 500M;
    location / {
            set $upstream localhost:5001;
            proxy_pass https://localhost:5001/;
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection keep-alive;
            proxy_set_header X-Forwarded-Proto $scheme;                
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Ssl on;
            proxy_set_header X-Forwarded-Proto https;
            proxy_read_timeout 900;
            proxy_request_buffering off;
            proxy_cache off;
            proxy_buffering off;
    }

    ssl_certificate /your/path/to/fullchain.pem;
    ssl_certificate_key /your/path/to/privkey.pem;
}

In most cases the settings are self explaining, but what you may particularly be interested in is the settings to the cache and buffering.

My general recommendation would be to stay with the original solution as long as it’s possible without putting any reverse proxy between registry server and the client. There are plenty of good to go solutions available out of the box:

• GitLab Container Registry
• Docker Registry Server
• Harbor

Even more are available online as a service.

I hope this information was helpful to you.
If you have some good private docker registry solution or you have any question don’t hesitate to drop me a message. You can find my contact channels below.

Take care,
Ievgen