Running a docker registry (v2) on Kubernetes is well documented as an addon to Kubernetes.
That setup, however, involves proxying the registry as
localhost on each Kubernetes node. While this simplifies pulling on nodes (no insecure registry issue, as it is localhost), this makes building and pushing outside the Kubernetes cluster unnecessarily complex and hacky (you need to
kubectl port-forward to access the registry, and you also must build your images with the tag like
localhost:5000/repository/image:version.) Moreover, it is based on Persistent Volume storage.
For a better docker registry setup, we wanted two things:
- S3 backed registry so that storage is managed better.
- Proper service for registry so that push and pull are more sane, and image tags are proper. We would like to push and pull from local workstation and our CI boxes. Also, at any time we can move to a different hosting solution for our private registry without have to retag and push images.
For S3 storage, we can utilize the ability to override all the configuration for the registry via environment variables. Our
ReplicationController looks like the following:
It is important to set
S3 so that the default storage configuration is overridden. If this is not done, you will get an error regarding multiple storage drivers.
REGISTRY_HTTP_SECRET has been added so that load balancing across multiple pods will work, when needed. Rest of the settings are pretty standard for a S3 backed registry, as per the docs.
We have a service that looks like below (For context, our Kubernetes cluster is on AWS, and has AWS aware features enabled):
We have a nice Route53 alias for the resulting ELB so that we can push and pull like we would to any other private registry. With the DNS name and S3 storage, moving away from Kubernetes for the registry is trivial too.
Improvements: Obviously, we are running an insecure registry at the moment. That’s something on our TODO list of things to fix. Currently, our CoreOS nodes, local workstations and CI boxes have Docker service running with the