Avoid Docker Hub pull limit with Gitlab

If you are familiar with Docker, you most likely know Docker Hub. For the others, a quick definition: Docker Hub is a service for finding and sharing container images. It is the world largest container image repository. It is widely used around the world

If you use Docker to test automatically your projects (CI/CD), maybe you encountered one issue: you exceed the pull limit allowed without a paid account. This limit is fixed to 100 pulls every 6 hours for anonymous pull (without account) and 200 pulls/6 hours for free accounts.

Gitlab provides a feature that allows us to "by-pass" this limitation. Actually, it is not a by-pass, it is a system that is more efficient than performs a pull every test.

This feature is called the Dependency Proxy. Concretely, Gitlab will check locally if a copy of the image is stored on your instance, if not, Gitlab will pull the image from Docker Hub and store it on your server. Thanks to that, a pull is performed only if you use a new image or if it is not the latest version of it.

Set up Dependency Proxy

First of all, the Dependency Proxy must be enabled on your Gitlab instance by your Administrator.

Then, you have to enable the Dependency Proxy for groups. It is not possible to enable it for a specific project only. Go to your Group settings -> Packages & Registries -> Dependency Proxy

Enable the Dependency Proxy On this page, you have the Dependency Proxy URL. This URL will be necessary for the next steps. Now, you can log to your personal Docker Hub:

docker login --username your_username --password your_password your_gitlab_instance.com

To pull an image (and store it in the Dependency Proxy if it does not exist yet):

docker pull https://your_gitlab_instance/your_group_dependency_proxy/containers/alpine:latest

The previous command will pull the alpine image with the latest tag

Dependency Proxy in CI/CD

It is of course possible to use the Dependency Proxy directly in CI/CD and .gitlab-ci.yml file.

First, we create a custom CI/CD variable. This variable can be created for a specific project, a group, or a complete Gitlab instance. In your project setting (or group settings), go to CI/CD

Expand the Variables section and click to add a new one:

Select a File variable type with the name DOCKER_AUTH_CONFIG and fill the body with:

{
    "auths": {
        "https://your_gitlab_instance/your_group/dependency_proxy/containers:443": {
            "auth": "SXRJc05vdEFSZWFsUGFzc3dvcmRVc2VybmFtZQo="
        }
    }
}

The string used in the auth field is actually the Base64 conversion of your username:password. For example, for the user UserTest with the password ThisIsMyPassword, the authentication string will be produced by the following command:

$ echo UserTest:ThisIsMyPassword | base64
VXNlclRlc3Q6VGhpc0lzTXlQYXNzd29yZAo=

After that, you are able to use your Dependency Proxy directly from a .gitlab-ci.yml file.

test:pytest:
  image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/python:3.9
  script:
    - pip install pytest --quiet
    - pytest tests

The variable ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX} contains the Dependency Proxy URL and Gitlab will use automatically your new DOCKER_AUTH_CONFIG variable to connect and pull the image.

For more information about the Dependency Proxy you can visit the Gitlab Documentation or this post

For personal or small team projects, this set-up should be enough to avoid the Docker Hub pull limit most of the time. For bigger projects, you should use a paid Docker Hub account (5000 pulls every 24 hours).

This blog post is licensed under CC BY-SA 4.0