App-of-Apps Pattern and Multi-Service Orchestration

The Failure

The platform team managed 15 ArgoCD Application resources manually. Adding a new service required creating the Application YAML, committing it, and applying it with kubectl. When the team standardized on a new sync policy, they had to update 15 files. When they moved to a new Git repository, they updated 15 repoURL fields. One was missed. That service stopped syncing for a week before anyone noticed.

The App-of-Apps pattern creates a single root Application that manages all other Applications. Adding a service means adding a directory. Changing a sync policy means changing the template.

The Mechanism

App-of-Apps Architecture

Root Application (watches ecommerce-infra/argocd/)
├── Application: checkout-service-production
├── Application: catalog-service-production
├── Application: inventory-service-production
├── Application: payments-service-production
└── Application: frontend-shell-production

The root Application watches a directory containing Application manifests. When a new manifest is added, ArgoCD creates the child Application. When a manifest is deleted, ArgoCD deletes the child Application (with prune: true).

ApplicationSet Alternative

ApplicationSet is a CRD that generates Application resources from templates. Instead of writing one Application per service, write one ApplicationSet that generates Applications for every directory in the apps/ path.

The Implementation

Root Application

# HARDENED: Root application managing all e-commerce apps
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ecommerce-root
  namespace: argocd
spec:
  project: ecommerce
  source:
    repoURL: https://github.com/acme/ecommerce-infra.git
    targetRevision: main
    path: argocd/production
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Child Application Template

# argocd/production/checkout-service.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-service-production
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: ecommerce
    environment: production
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ci-alerts
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: ecommerce
  source:
    repoURL: https://github.com/acme/ecommerce-infra.git
    targetRevision: main
    path: apps/checkout-service/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

ApplicationSet for All Services

# HARDENED: ApplicationSet generates apps from directory structure
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ecommerce-services
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/acme/ecommerce-infra.git
        revision: main
        directories:
          - path: apps/*

  template:
    metadata:
      name: "{{path.basename}}-production"
      namespace: argocd
      labels:
        app.kubernetes.io/part-of: ecommerce
        environment: production
      annotations:
        notifications.argoproj.io/subscribe.on-sync-failed.slack: ci-alerts
    spec:
      project: ecommerce
      source:
        repoURL: https://github.com/acme/ecommerce-infra.git
        targetRevision: main
        path: "{{path}}/overlays/production"
      destination:
        server: https://kubernetes.default.svc
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
        retry:
          limit: 5
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m

Multi-Environment ApplicationSet

# HARDENED: Generate apps for all services × all environments
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ecommerce-all
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/acme/ecommerce-infra.git
              revision: main
              directories:
                - path: apps/*
          - list:
              elements:
                - environment: dev
                  autoSync: "true"
                  prune: "true"
                - environment: staging
                  autoSync: "true"
                  prune: "true"
                - environment: production
                  autoSync: "true"
                  prune: "true"

  template:
    metadata:
      name: "{{path.basename}}-{{environment}}"
      namespace: argocd
      labels:
        app.kubernetes.io/part-of: ecommerce
        environment: "{{environment}}"
    spec:
      project: ecommerce
      source:
        repoURL: https://github.com/acme/ecommerce-infra.git
        targetRevision: main
        path: "{{path}}/overlays/{{environment}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{environment}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

With this ApplicationSet, adding a new service is a single action: create a directory under apps/ with the standard Kustomize structure. The ApplicationSet generates dev, staging, and production Applications automatically.

The Gate

The root Application monitors all child Applications. If any child is Degraded or OutOfSync, the root Application reflects that status. The ArgoCD dashboard shows the health of the entire platform in one view.

Adding a new Application via the App-of-Apps pattern requires a PR to the infra repo. The PR review is the gate. Once merged, ArgoCD creates the Application and syncs it automatically.

The Recovery

Child Application is orphaned: The root Application’s prune: true setting handles this. If a child Application manifest is removed from Git, ArgoCD deletes the child Application and all its managed resources.

ApplicationSet generates incorrect apps: Check the generator configuration. The directories generator creates one application per directory. If a directory exists but has no valid Kustomize overlay, the generated Application will fail to sync. Add directory filtering to exclude non-service directories.

Need to temporarily disable a service: Do not delete the Application manifest. Set syncPolicy: {} (empty, disabling auto-sync) and set the Application’s operation.sync.dryRun: true. This keeps the Application visible but stops it from syncing.