GitOps and multi-repo versioning

Multiple Git repositories/apps, one Kubernetes (K8s) deployment. What do you do to streamline the development process?

This time, I chose GitOps, a declarative approach where Git is the single source of truth for the current deployment state. How many replicas of app A are running? What version of app B? Just Git it.

While setup went mostly smoothly, one challenge was versioning. What commit from what repository triggered an update to the K8s deployment? What was the deployment version two commits ago? How to automate all this? We’ll tackle these questions and more, but first a little background.

Background

Above outlines the basic configuration. There are multiple app repositories that collectively compose the platform. The platform config repository holds the K8s config yaml files that declare what the K8s deployment should look like (i.e. which version of app A, how many replicas, etc). The GitOps controller lives in the same K8s cluster and applies any config changes to the K8s deployment.

The diagram above details the development workflow. With Cloud Build’s help, pushing a commit to master in any app repository triggers a Docker image build. This push then also triggers an update — via kustomize — to the app’s new image version in a K8s config yaml file in the platform config repository. Finally, the controller updates the K8s deployment to reflect the new configuration.

Alright, now the fun part: versioning.

Versioning

With versioning, we want to address two issues: logging which commit from which app repository triggered the config update and versioning each update in sequential order — both happening within the platform config repository. For this, we will use Git tags and semantic versioning. More specifically, we will tag each commit twice.

The first app-api-9d4af7c-rc tag tells us that this commit that updated the yaml config was triggered by commit 9d4af7c in the app-api repository (rc stands for release candidate). The second tag gives us the semantic version for the deployment config’s state in this commit plus the commit’s short SHA-1 hash for reference. If the last commit was version 0.7.15–19, then the next commit would be 0.7.15–20 and so on.

Alright, so how do we automate this tagging process? Within a Cloud Build config file, it might look like this.

The execution of this Cloud Build config file that rests in each app repository is triggered on a push to master. During its execution, we have access to built-in values such as $REPO_NAME (which would translate to app-api above). We can also substitute our own custom values by placing an underscore before the variable name (such as $_CONFIG_REPO_URI).

First we clone the platform config repository, configure the Git user, update the image’s version via kustomize, and then commit. Next, we create two tags, one to log where the commit is originating from, and the other to add the current semantic version based off of the latest commit in the platform config repository. And finally, we push, taking care to only include annotated tags.

So what’s with the --exclude=”*-rc” in the second tag? According to git-scm.com:

If the tag points to the commit, then only the tag is shown. Otherwise, it suffixes the tag name with the number of additional commits on top of the tagged object and the abbreviated object name of the most recent commit.

In addition to the x.x.x-x-(commit sha)-rc tags which represent staging environment deployments, we also have production environment tags in x.x.x format not shown. We exclude the staging version tags that end with *-rc because we want to build all staging versions off of the latest production version, which in this case is 0.7.15. After the next production commit, say tagged with 0.8.0, all subsequent staging commits will become 0.8.0-xx-(commit sha)-rc.

Using this tagging system, it is now easy to trigger a new staging or production deployment based on the tag’s format.

Conclusion

With so many CI/CD options out there, choosing and configuring one can be a daunting task. I hope you found something useful in this approach and encourage you to give GitOps a try.