An End-to-End GitOps Automation with ArgoCD and Jenkins on Kubernetes — Part I — Why Bother with ArgoCD?
In the recent four-story-series of Building CI/CD Pipeline with Jenkins Exclusively on Kubernetes, I presented two blueprint projects (a Django and a Java web app) on Kubernetes managed only by Jenkins pipelines (Part I, Part II, Part III and Part IV). This approach solves a majority of problems in CI/CD lifecycle such that the journey started from developer’s local PC to the end of pull request/merge to master branch is well defined on Jenkins platform. However, this is just the Continuous Integration phase that is being managed by Jenkins.
To complete the picture of CI/CD, continuous delivery/deployment was largely handled manually in the last series. This was not a complete manual process but workload manifests (pod, statefulset, deployment etc.) was parameterized within Jenkins pipeline and triggered from there with kubectl tool.
The CD approach taken in the last series is called “push based”. This is because, you use the CI/CD tool to push manifests to K8S within pipeline. The other option is the “pull based” such that ArgoCD (or another GitOps tool) polls version control system to check for changes. A full comparison between two approaches is avoided here, however the main disadvantage of the push based approach is that you need to expose K8S credentials to CI/CD tool to be able to allow it to make modifications. An example usage can be seen in below visual from my previous series. A neat discussion on pull and push based CD can be found here.
To summarize in the simplest terms, added value of this series will be the automation of CD phase with the help of ArgoCD. This one sentence goal corresponds to major effort such as integrating ArgoCD into the pipeline. But why do we need this change? The pipelines in the last series worked perfectly fine and the whole process were managed on Kubernetes without an issue. There are two main points to justify this. First is the scale. If you’re managing, lets say, two K8S clusters and a total of 20 applications, you can manage Kubernetes manifests manually. However, today’s world is the microservices’ (even nanoservices) and things scale up really fast. The public grafana dashboard of an e-commerce company shows that you can find yourself in a position where more than 350 K8S clusters and 14000 microservices are waiting to be managed. Of course this is an outlier example but, even much smaller configurations are too risky to be handled manually in enterprise level. The second one is the need to version control of K8S objects. As of now, there are more than 60 different types of native K8S objects. Multiply it with the namespace and cluster count and the result would be in thousands scale in any enterprise. Moreover, you would not be the only person who is managing this environment but a member of a team. Imagine the chaos of five people managing a pool of thousands of objects without a version control system. These two points were extended to ten in a sound blog post here.
To keep things manageable, this series will be comprised of three stories. In the first one, I’ll try to justify the usage of ArgoCD and make a summary of installation and configuration on K8S. In the second one, I’ll present a simple java project’s deployment with Jenkins and ArgoCD on a K8S cluster. And in the last one, I’ll focus on secret management in ArgoCD with Hashicorp Vault’s integration. Similar to previous series about Jenkins, this one will not be a step-by-step guide requiring any background knowledge. On the contrary, DevOps, Linux containers, Kubernetes, Git, Jenkins and ArgoCD basic knowledge is needed for a better understanding. The main goal is to discuss and summarize the approach taken to achieve a completely automated and version controlled CI/CD cycle.
Installing ArgoCD, official documentation is here to help. There are four different options regarding to high-availability (HA) and level of permissions that Argo uses. The best practice in a large enterprise should be installing Argo in an isolated cluster with HA. This is because Argo will be in the center of all CD operation and must not be affected from failures of any K8S cluster-level outages due to a co-hosted application’s malfunction. It also should have namespace-level privileges to follow the “least privileges” principle. Because I have only one cluster in this series, HA and non-namespace-level installation path was chosen. Installation is as easy as executing the below with kubectl.
With a successful execution, you will have a long list of components as it can be seen below. Looking in and trying to understand the manifest of workload objects (statefulsets and deployments) would be useful after installation for a better comprehension of the system as a whole. Argo docs have some useful pages to explain each components role.
To manage Argo, you have two options. Web UI and CLI. Throughout this series, Web UI will be used to stay illustrative. Being a CLI type, I personally find Argo’s Web UI quite successful such that except some specific needs, it is more than enough for an admin. Something like below will be greeting you after installation.
Argo creates its own ecosystem with components within. There are two important objects to mention. The first and foremost one is the “application”. This is the basic building block of Argo. Every deployment you’ll make is an application in Argo’s perspective. With installation, it creates a Custom Resource Definiton (CRD) (I know you love them K8S admins :)) with the name “application”. Below is a sample application’s manifest. Usually you would want to manage applications from web UI or CLI but knowing what it corresponds within K8S is needed for DevOps admins.
The second important component is the “project”. Argo created it to group applications for easier management. Every Argo installation comes with a default “project” and you can perfectly live with it without creating another one. Multiple projects can be created to limit the use of source repositories or destination clusters. As it can be seen below, default project comes with no such limitations.
Lets go on with the mindset of using Argo. Argo depends on version control systems (VCS) such as Git to manage objects on K8S. For Argo to work in your environment as desired, VCS must be the only source for K8S object modification. In other words, you must stop the use of kubectl or any other K8S client to manage applications running on K8S. They call it the “Single Source of Truth”.
Another important concept is with a fancy name of “Reconciliation Loop”. As explained above, after the adoption of Argo, no K8S sources must be modified with kubectl. All the modifications must be via Git Commits. But, how Argo will know about this changes? A notable delay between the changes in VCS and their reflection on K8S would make things unmanageable thus, this is a major problem to solve.
The first option is to use “Reconciliation Timeout”. This is a value in argocd-cm configmap to be sent to argocd-repo-server deployment manifest which tell argocd to check VCS for changes in every X seconds. Although setting this to 1 second is theoretically possible but in my personal experience, this always introduces a delay no matter how low it is set.
The second and more widespread option comes from VCS platforms. Personal choice of mine will be GitHub in this series. Webhooks comes into play to poke argocd-server service every time a commit is made into a specific branch in a repository. Sample configuration is given below for reference.
As the result of webhook creation, any commit that is made to a branch in GitHub instantly triggers to the related Argo application and makes the necessary changes. It can be seen below in a sample video.
Another major component of Argo is the health check mechanism. By default, every object comes with its own default health check logic by Argo. Little green heart icons in below visual represents the “healthy” state of each object. Progressing, Degraded, Missing and Unknown are the other predefined states.
However, health check mechanism of some K8S objects are not perfect. Therefore, some of them may spoil the overall health state of an application. Untrue triggering of non-healthy state of an application may cause unwanted consequences such as false positive Discord or Slack notifications or DevOps engineers being wake up at 3:00 AM. As a solution to such problems, Argo enables the creation of custom health checks. Lua scripting language is the tool for writing them. As a common example, Nginx ingress object has an incompatibility with default health check logic of Argo discussed here. You need to create a custom health check script (given below) and add it in argo-cm configmap in order to present the correct health information.
This is it for installation, configuration and some insights about Argo’s components. In the next story, there will be a sample java web application being go through CI/CD pipeline with Jenkins and Argo. See you there!
Links to other stories in this series.
PS: If you liked the article, please support it with claps to reach more people. Cheers!
