GitOps with ArgoCD

GitOps is a single principle: the desired state of the system is defined in Git, and an operator continuously reconciles the live state to match. If someone manually changes a Kubernetes resource, the operator reverts it. If someone pushes a change to Git, the operator applies it. Git is the source of truth. The cluster is a reflection.

ArgoCD implements this principle for Kubernetes. It watches Git repositories, compares manifests in Git with resources in the cluster, and syncs the differences.

The Failure

The team deployed five services using kubectl apply from a CI pipeline. Each service had its own deployment script. Over six months, the scripts diverged: some used kubectl apply, others used kubectl replace, one used helm upgrade. When a deployment failed, the engineer would SSH into the bastion host and run kubectl commands manually to fix the state. After a year, nobody knew what was actually running in production. The cluster state had drifted from every Git repository.

ArgoCD eliminates this drift. The cluster state matches Git. If it does not, ArgoCD shows the diff and optionally self-heals.

The Mechanism

The Reconciliation Loop

1. ArgoCD polls the Git repository every 3 minutes (configurable, or webhook-triggered)
2. ArgoCD renders the manifests (Helm template, Kustomize build, plain YAML)
3. ArgoCD compares rendered manifests with live cluster state
4. If they differ, ArgoCD reports the application as OutOfSync
5. If automated sync is enabled, ArgoCD applies the diff
6. ArgoCD monitors the rollout and reports health status

Application States

State	Meaning	Action
Synced + Healthy	Git = Cluster, all pods ready	None
OutOfSync	Git ≠ Cluster	Sync (automatic or manual)
Synced + Degraded	Git = Cluster, but pods unhealthy	Investigate pod health
Synced + Progressing	Sync applied, rollout in progress	Wait
Unknown	ArgoCD cannot determine state	Check connectivity

The Implementation

ArgoCD Installation

# HARDENED: ArgoCD installation with resource limits
apiVersion: v1
kind: Namespace
metadata:
  name: argocd
---
# Install ArgoCD with Kustomize
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.0/manifests/install.yaml
patches:
  - target:
      kind: Deployment
      name: argocd-server
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/resources
        value:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi

Application for the Checkout Service

# HARDENED: ArgoCD Application with full configuration
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-service
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: ecommerce
    team: checkout
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: ecommerce

  source:
    repoURL: https://github.com/acme/ecommerce-infra.git
    targetRevision: main
    path: apps/checkout-service/overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true # Delete resources removed from Git
      selfHeal: true # Revert manual cluster changes
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas # Allow HPA to manage replicas

ArgoCD Project for E-Commerce

# HARDENED: Project scoping for multi-team access
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: ecommerce
  namespace: argocd
spec:
  description: E-commerce platform services
  sourceRepos:
    - https://github.com/acme/ecommerce-infra.git

  destinations:
    - server: https://kubernetes.default.svc
      namespace: dev
    - server: https://kubernetes.default.svc
      namespace: staging
    - server: https://kubernetes.default.svc
      namespace: production

  clusterResourceWhitelist:
    - group: ""
      kind: Namespace

  namespaceResourceWhitelist:
    - group: ""
      kind: "*"
    - group: apps
      kind: "*"
    - group: batch
      kind: "*"
    - group: networking.k8s.io
      kind: "*"

  roles:
    - name: developer
      description: Read-only access, sync dev/staging
      policies:
        - p, proj:ecommerce:developer, applications, get, ecommerce/*, allow
        - p, proj:ecommerce:developer, applications, sync, ecommerce/*-dev, allow
        - p, proj:ecommerce:developer, applications, sync, ecommerce/*-staging, allow
    - name: platform
      description: Full access to all environments
      policies:
        - p, proj:ecommerce:platform, applications, *, ecommerce/*, allow

Webhook for Immediate Sync

# GitHub webhook triggers ArgoCD sync instead of waiting for poll
# Configure in GitHub repo settings → Webhooks
# URL: https://argocd.acme.com/api/webhook
# Content type: application/json
# Secret: $ARGOCD_WEBHOOK_SECRET
# Events: Push

The Gate

ArgoCD’s health check is the deployment gate. After a sync, ArgoCD monitors the rollout. If pods fail to become ready within the progressDeadlineSeconds (default 600s), ArgoCD marks the application as Degraded. Notifications are sent to the team’s Slack channel.

The selfHeal: true setting prevents manual changes from persisting. If someone runs kubectl scale deployment checkout-service --replicas=1 in production, ArgoCD detects the drift and reverts to the replica count defined in Git.

The Recovery

Application stuck in Progressing: The rollout is taking too long. Check pod events: kubectl -n production describe pod -l app=checkout-service. Common causes: image pull errors, insufficient resources, failing readiness probes.

Application shows OutOfSync but nothing changed in Git: ArgoCD detects a difference between rendered manifests and live state. Use argocd app diff checkout-service to see the exact diff. Common cause: a mutating webhook or admission controller modified the resource after ArgoCD applied it. Use ignoreDifferences to exclude those fields.

Need to rollback: In ArgoCD UI, click “History and Rollback” → select the previous successful sync → click “Rollback.” Or revert the Git commit and let ArgoCD sync the revert.