Clusters as Cattle

GitOps for Clusters

Wander Boessenkool

Problem

Maintaining Multiple OpenShift/K8S Clusters

Installing is not the hard part

Challenges

  • Known State
    •
  • Disaster Recovery
    •
  • Multi-Cluster
    •

Click-Ops Won’t Do

Everything as Code

Bring On ArgoCD

(OpenShift GitOps)

How?

  • Single Application?

    •

  • Multiple Applications?

    •

  • App of Apps?

    •

  • ApplicationSet?

    •

Copy-Pasta Between Clusters?

NO!

Kustomize with Unified Base

What if I want to update my Base?

Versioned Bases

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- git@gitlab.com/org/repo.git//path?ref=TAG

What About Dependencies?

(Configuration) Apps Should be Stand-Alone

Where Possible

But ArgoCD will retry otherwise

Permissions

Out-of-the-Box ArgoCD can't do everything

Making ArgoCD Cluster Admin is NOT an option

One App with common Roles and RoleBindings

  • CRDs

    •

  • MachineConfigs

    •

  • SCCs

    •

  • Etc.

    •

Included per App

Where Needed

Resource Ordering

Namespace before OperatorGroup

OperatorGroup before Subscription

Etc.

ArgoCD Sync-Waves

metadata:
  name: hope24
  annotations:
    argocd.argoproj.io/sync-wave: "-10"

Missing Resource Types

apiVersion: some.new.group/v1
kind: definedByNewOperator
metadata:
  annotations:
    argocd.argoproj.io/sync-options: |
      SkipDryRunOnMissingResource=true

Secrets

Three may keep a secret, if two of them are dead.

Benjamin Franklin

Choices

  • SealedSecrets
    •
  • ExternalSecrets Operator
    •
  • Other Secret Providers
    •
  • NEVER directly in Git
    •

Automation

Ansible

  • Install and Configure ArgoCD
    •
  • Setup Repositories
    •
  • Configure Secret Management
    •
  • Configure Applications in Git
    •
  • Manage Promotion of Bases
    •
  • Check for Drift
    •

HCS OpenShift Install

openshift_config_gitops.yml

What’s Next?

Keep Your ClueBat Closeby

At least until everybody stops doing ClickOps

Implement Four Eyes

Everything becomes a PR

No matter how trivial

Regular Reflection

  • Moving manifests between Bases and Overlays
    •
  • Way of Working
    •
  • Preparing for future changes
    •

Lifecycle Management

Separate Dev and Test/Staging clusters

Cluster Upgrades via GitOps?

Try it, not as fun as you think

Questions?