Introduction

In this post, I describe how to configure my private Kubernetes cluster to authenticate against a private Docker registry to pull images using a Docker config json secret resource.

I have a private kubernetes (k8s) cluster running on my homelab server rack. I also host a private Docker registry to store images used by pods within my k8s cluster. Even though my k8s cluster and docker registry are private, I still prefer to secure access to my docker registry with a username and password. For the most part, this is not difficult, but extra whitespace when base64 encoding turned out to be a tricky problem to solve.

Overview

The process to configure private docker registry access is as follows

Generate credentials for registry access, if you haven't already
Generate a custom docker config.json file containing base64 encoded credentials
Create Kuberenetes generic secret from custom config.json
Deploy test pod to pull image from private registry

Beware, it is very important to avoid extra whitespace and line breaks when base64 encoded this data. If you fail to do this, your pods will throw errors indicating they cannot access the docker registry.

Generate Docker Registry credentials

I use GitLab for source code management and, in particular, the docker registry feature is what I use to host images for my projects. Using the GitLab web UI, I generate a read-only deploy token for the docker registry. For purposes of this article, here are fake credentials

username: gitlab+deploy-token-fake
password: aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1kUXc0dzlXZ1hjUQ==

Generate Custom Docker Configuration

When authenticating against docker, I use my operating systems credential helper to securely store my username/password. The version of docker I'm using by default will automatically locate and use this credential store. Extra steps are necessary to avoid using this credential store and base64 encode the credentials in the config.json file.

To start, use the --config option of docker to specify a custom directory to store the config.json file

$ docker --config ./docker-k8s login gitlab.homelab:5555 -u gitlab+deploy-token-fake
Password: 
Login Succeeded

Here is the breakdown of that command

docker is the docker command line tool
--config ./docker-k8s specifies a custom configuration directory containing a new and different config.json file separate from the default (~/.docker/config.json), in this case a new directory docker-k8s in the current directory
login is docker's login subcommand
gitlab.homelab:5555 is my GitLab server's specified container registry endpoint
-u gitlab+deploy-token-fake uses dockers login command option -u for specifying a user, in this case gitlab+deploy-token-fake

If the directory used with the --config option does not exist, it will be created

$ ls
docker-k8s

$ ls docker-k8s/
config.json

If you check the contents of docker-k8s/config.json, you'll see an entry was added for gitlab.homelab:5555 server

$ cat docker-k8s/config.json 
{
	"auths": {
		"gitlab.homelab:5555": {}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.5 (darwin)"
	},
	"credsStore": "osxkeychain"
}

Notice, however, no base64 encoded credentials are in this file. This is because the credsStore option specifies osxkeychain which is my default credential store on macOS.

The docker version I'm using here does not have an option to override the credential store while using the docker login subcommand, so I have to manually edit the file to remove the credsStore option and replace it with the credHelper configuration

...
	"credHelpers": {
		"gitlab.homelab:5555": ""
	}
...

This will ensure that osxkeychain is not used while logging in.

The new docker-k8s/config.json file looks like this

$ cat docker-k8s/config.json 
{
	"auths": {
		"gitlab.homelab:5555": {}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.5 (darwin)"
	},
	"credHelpers": {
		"gitlab.homelab:5555": ""
	}
}

I next re-run the login command to store my credentials as a base64 encoded string. The command warns you that these are insecurely stored in a file

$ docker --config ./docker-k8s login gitlab.homelab:5555 -u gitlab+deploy-token-fake
Password: 
WARNING! Your password will be stored unencrypted in docker-k8s/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

The final version of the docker-k8s/config.json looks like this

$ cat docker-k8s/config.json 
{
	"auths": {
		"gitlab.homelab:5555": {
			"auth": "Z2l0bGFiK2RlcGxveS10b2tlbi1mYWtlOmFIUjBjSE02THk5M2QzY3VlVzkxZEhWaVpTNWpiMjB2ZDJGMFkyZy9kajFrVVhjMGR6bFhaMWhqVVE9PQ=="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.5 (darwin)"
	}
}

If I decode the base64 encoded string in the auth entry, I see my GitLab deploy token and password

$ echo "Z2l0bGFiK2RlcGxveS10b2tlbi1mYWtlOmFIUjBjSE02THk5M2QzY3VlVzkxZEhWaVpTNWpiMjB2ZDJGMFkyZy9kajFrVVhjMGR6bFhaMWhqVVE9PQ==" | base64 --decode
gitlab+deploy-token-fake:aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1kUXc0dzlXZ1hjUQ==

This docker-k8s/config.json can be used to create a Kubernetes docker config secret.

Shortcut to Create Custom Docker Config

Yes, I know. It is cruel to tell you about this shortcut after I demonstrated the long way. But it's hard to appreciate a shortcut unless you experience the difficulty firsthand. Nonetheless, here is a quick way you can generate the file in a couple of steps.

The following command is a actually two commands run together. The first creates a custom docker config directory, the second creates a special docker config.json file that already already has empty credhelper value.

Be sure to replace gitlab.homelab:5555 with your docker registry URL.

$ mkdir docker-k8s-shortcut && echo "{\"credHelpers\":{\"gitlab.homelab:5555\": \"\"}}" > docker-k8s-shortcut/config.json

You can now simply run the docker login command without having to edit the config file directly

$ docker --config ./docker-k8s-shortcut login gitlab.homelab:5555 -u gitlab+deploy-token-fake
Password: 
WARNING! Your password will be stored unencrypted in docker-k8s-shortcut/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

$ cat docker-k8s-shortcut/config.json 
{
	"auths": {
		"gitlab.homelab:5555": {
			"auth": "Z2l0bGFiK2RlcGxveS10b2tlbi1mYWtlOmFIUjBjSE02THk5M2QzY3VlVzkxZEhWaVpTNWpiMjB2ZDJGMFkyZy9kajFrVVhjMGR6bFhaMWhqVVE9PQ=="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.5 (darwin)"
	},
	"credHelpers": {
		"gitlab.homelab:5555": ""
	}
}

Create a Kubernetes Secret

Next, I create a Kubernetes secret resource from the docker config json fil. All nodes within my cluster can then use the secret.

$ kubectl create secret generic gitlab-registry-cred --from-file=.dockerconfigjson=./docker-k8s-shortcut/config.json --type=kubernetes.io/dockerconfigjson

Here is a breakdown of the command

kubectl create secret generic is the k8s command line command to create a secret resource in which you can store the docker config json information
gitlab-registry-cred is simply the name I chose for the secret resource; any value for the name will do here
--from-file=.dockerconfigjson= is the command line option to create the secret resource from a docker config json file
./docker-k8s-shortcut/config.json is the special docker config file we created in the previous section
--type=kubernetes.io/dockerconfigjson is the command line option to specify the type of secret I am creating

I can now use gitlab-registry-cred as the imagePullSecret in my pod specs.

Deploy a Test Pod

Finally, I deploy a test pod to ensure the secret credential works. I use the following test.yaml file which sets up a test pod to pull from my private docker registry gitlab.homelab:5555/testrepo/testwebapp:v1. Here, the reader should assume I've successfully pushed a docker built image with a v1 label to the private repo.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment 
  labels:
    app: test-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
      - image: gitlab.homelab:5555/testrepo/testwebapp:v1
        name: test-app
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: gitlab-registry-cred

Notice the imagePullSecrets names the gitlab-registry-cred we created in the previous section. Check the status of your pods to see they have successfully been pulled.

$ kubectl get pods | grep  test-deployment
test-deployment-67f775597-hzfmf   1/1     Running   0          2m
test-deployment-67f775597-w5k4m   1/1     Running   0          2m
test-deployment-67f775597-zvnvj   1/1     Running   0          2m

And you're done!