If you found value in this post, consider following me on X @davidpuplava for more valuable information about Game Dev, OrchardCore, C#/.NET and other topics.
In this post, I describe how to configure my private Kubernetes cluster to authenticate against a private Docker registry to pull images using a Docker config json secret resource.
I have a private kubernetes (k8s) cluster running on my homelab server rack. I also host a private Docker registry to store images used by pods within my k8s cluster. Even though my k8s cluster and docker registry are private, I still prefer to secure access to my docker registry with a username and password. For the most part, this is not difficult, but extra whitespace when base64 encoding turned out to be a tricky problem to solve.
The process to configure private docker registry access is as follows
Beware, it is very important to avoid extra whitespace and line breaks when base64 encoded this data. If you fail to do this, your pods will throw errors indicating they cannot access the docker registry.
I use GitLab for source code management and, in particular, the docker registry feature is what I use to host images for my projects. Using the GitLab web UI, I generate a read-only deploy token for the docker registry. For purposes of this article, here are fake credentials
username: gitlab+deploy-token-fake
password: aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1kUXc0dzlXZ1hjUQ==
When authenticating against docker, I use my operating systems credential helper to securely store my username/password. The version of docker I'm using by default will automatically locate and use this credential store. Extra steps are necessary to avoid using this credential store and base64 encode the credentials in the config.json file.
To start, use the --config
option of docker
to specify a custom directory to store the config.json
file
$ docker --config ./docker-k8s login gitlab.homelab:5555 -u gitlab+deploy-token-fake
Password:
Login Succeeded
Here is the breakdown of that command
docker
is the docker command line tool--config ./docker-k8s
specifies a custom configuration directory containing a new and different config.json
file separate from the default (~/.docker/config.json), in this case a new directory docker-k8s
in the current directorylogin
is docker's login subcommandgitlab.homelab:5555
is my GitLab server's specified container registry endpoint-u gitlab+deploy-token-fake
uses dockers login command option -u
for specifying a user, in this case gitlab+deploy-token-fake
If the directory used with the --config
option does not exist, it will be created
$ ls
docker-k8s
$ ls docker-k8s/
config.json
If you check the contents of docker-k8s/config.json
, you'll see an entry was added for gitlab.homelab:5555
server
$ cat docker-k8s/config.json
{
"auths": {
"gitlab.homelab:5555": {}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.5 (darwin)"
},
"credsStore": "osxkeychain"
}
Notice, however, no base64 encoded credentials are in this file. This is because the credsStore
option specifies osxkeychain
which is my default credential store on macOS.
The docker version I'm using here does not have an option to override the credential store while using the docker login
subcommand, so I have to manually edit the file to remove the credsStore
option and replace it with the credHelper
configuration
...
"credHelpers": {
"gitlab.homelab:5555": ""
}
...
This will ensure that osxkeychain
is not used while logging in.
The new docker-k8s/config.json
file looks like this
$ cat docker-k8s/config.json
{
"auths": {
"gitlab.homelab:5555": {}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.5 (darwin)"
},
"credHelpers": {
"gitlab.homelab:5555": ""
}
}
I next re-run the login command to store my credentials as a base64 encoded string. The command warns you that these are insecurely stored in a file
$ docker --config ./docker-k8s login gitlab.homelab:5555 -u gitlab+deploy-token-fake
Password:
WARNING! Your password will be stored unencrypted in docker-k8s/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
The final version of the docker-k8s/config.json
looks like this
$ cat docker-k8s/config.json
{
"auths": {
"gitlab.homelab:5555": {
"auth": "Z2l0bGFiK2RlcGxveS10b2tlbi1mYWtlOmFIUjBjSE02THk5M2QzY3VlVzkxZEhWaVpTNWpiMjB2ZDJGMFkyZy9kajFrVVhjMGR6bFhaMWhqVVE9PQ=="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.5 (darwin)"
}
}
If I decode the base64 encoded string in the auth
entry, I see my GitLab deploy token and password
$ echo "Z2l0bGFiK2RlcGxveS10b2tlbi1mYWtlOmFIUjBjSE02THk5M2QzY3VlVzkxZEhWaVpTNWpiMjB2ZDJGMFkyZy9kajFrVVhjMGR6bFhaMWhqVVE9PQ==" | base64 --decode
gitlab+deploy-token-fake:aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1kUXc0dzlXZ1hjUQ==
This docker-k8s/config.json
can be used to create a Kubernetes docker config secret.
Yes, I know. It is cruel to tell you about this shortcut after I demonstrated the long way. But it's hard to appreciate a shortcut unless you experience the difficulty firsthand. Nonetheless, here is a quick way you can generate the file in a couple of steps.
The following command is a actually two commands run together. The first creates a custom docker config directory, the second creates a special docker config.json
file that already already has empty credhelper value.
Be sure to replace gitlab.homelab:5555
with your docker registry URL.
$ mkdir docker-k8s-shortcut && echo "{\"credHelpers\":{\"gitlab.homelab:5555\": \"\"}}" > docker-k8s-shortcut/config.json
You can now simply run the docker login
command without having to edit the config file directly
$ docker --config ./docker-k8s-shortcut login gitlab.homelab:5555 -u gitlab+deploy-token-fake
Password:
WARNING! Your password will be stored unencrypted in docker-k8s-shortcut/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ cat docker-k8s-shortcut/config.json
{
"auths": {
"gitlab.homelab:5555": {
"auth": "Z2l0bGFiK2RlcGxveS10b2tlbi1mYWtlOmFIUjBjSE02THk5M2QzY3VlVzkxZEhWaVpTNWpiMjB2ZDJGMFkyZy9kajFrVVhjMGR6bFhaMWhqVVE9PQ=="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.5 (darwin)"
},
"credHelpers": {
"gitlab.homelab:5555": ""
}
}
Next, I create a Kubernetes secret resource from the docker config json fil. All nodes within my cluster can then use the secret.
$ kubectl create secret generic gitlab-registry-cred --from-file=.dockerconfigjson=./docker-k8s-shortcut/config.json --type=kubernetes.io/dockerconfigjson
Here is a breakdown of the command
kubectl create secret generic
is the k8s command line command to create a secret resource in which you can store the docker config json informationgitlab-registry-cred
is simply the name I chose for the secret resource; any value for the name will do here--from-file=.dockerconfigjson=
is the command line option to create the secret resource from a docker config json file./docker-k8s-shortcut/config.json
is the special docker config file we created in the previous section--type=kubernetes.io/dockerconfigjson
is the command line option to specify the type of secret I am creatingI can now use gitlab-registry-cred
as the imagePullSecret in my pod specs.
Finally, I deploy a test pod to ensure the secret credential works. I use the following test.yaml
file which sets up a test pod to pull from my private docker registry gitlab.homelab:5555/testrepo/testwebapp:v1
. Here, the reader should assume I've successfully pushed a docker built image with a v1
label to the private repo.
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
labels:
app: test-app
spec:
replicas: 3
selector:
matchLabels:
app: test-app
template:
metadata:
labels:
app: test-app
spec:
containers:
- image: gitlab.homelab:5555/testrepo/testwebapp:v1
name: test-app
ports:
- containerPort: 80
imagePullSecrets:
- name: gitlab-registry-cred
Notice the imagePullSecrets
names the gitlab-registry-cred
we created in the previous section. Check the status of your pods to see they have successfully been pulled.
$ kubectl get pods | grep test-deployment
test-deployment-67f775597-hzfmf 1/1 Running 0 2m
test-deployment-67f775597-w5k4m 1/1 Running 0 2m
test-deployment-67f775597-zvnvj 1/1 Running 0 2m
And you're done!
If you found value in this post, consider following me on X @davidpuplava for more valuable information about Game Dev, OrchardCore, C#/.NET and other topics.