Kubernetes Secrets Management: 5 Best Practices You Need to Know

The Problem

Kubernetes secrets, by default, are base64-encoded in etcd, offering a false sense of security; they are easily decoded by anyone with cluster access. This practice exposes sensitive data and creates a significant vulnerability in your infrastructure.

Why This Matters

Storing credentials directly in Kubernetes manifests or using simple environment variables is a widespread anti-pattern. This exposes secrets to potential breaches, especially in multi-tenant environments or with compromised developer machines. The cost of a compromised secret can range from data loss and service disruption to significant financial penalties and reputational damage.

Key Insights

• Base64 encoding is not encryption: Kubernetes secrets are not encrypted by default, only encoded.
• External Secrets Operator: Simplifies integration with cloud provider secret management services.
• Sealed Secrets: Allows for safe storage of encrypted secrets in Git repositories, used by teams like Bitnami.

Working Example

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
data:
  username: <base64 encoded username>
  password: <base64 encoded password>

Practical Applications

• Netflix: Uses a custom secrets management system built on top of Vault to handle sensitive data across its microservices architecture.
• Pitfall: Storing secrets in environment variables – easily exposed through container logs and debugging tools.

References:

• https://dev.to/abinasharma001/kubernetes-secrets-management-5-best-practices-you-need-toknow-5ek4