How to Keep Your Cool When Your Kubernetes Cluster Crashes, or Why You Need Velero

Imagine this: it's Friday evening, you're planning to relax, when suddenly a Slack notification pops up — one of your production Kubernetes clusters has bitten the dust. A misconfiguration, a cloud provider outage, or just plain human error — the cause no longer matters. What matters is how quickly you can get everything back up and running. If you only have YAML files in Git, good luck recovering stateful applications and their data.

This is where Velero comes in. It's a project from VMware (originally created by Heptio) that has become the de facto standard for backup in the K8s world.

What It Is and Why GitOps Alone Isn't Enough

Many developers believe that if their entire infrastructure is defined in code (IaC) and deployed through ArgoCD or Flux, backups are unnecessary. This is a dangerous misconception. Git stores manifests, but it knows nothing about the state of your Persistent Volumes (PV), dynamically issued certificates, or secrets that never made it into the repository.

Velero does two things: it saves Kubernetes API objects (deployments, configmaps, secrets) and takes disk snapshots with data. It works in public clouds (AWS, GCP, Azure) as well as in on-premises data centers on bare metal.

Three Scenarios Where Velero Saves the Day

I often see Velero used not just for disaster protection. Here are the main use cases:

1. Classic Disaster Recovery. This one's straightforward: the cluster is dead, you spin up a new one, and restore everything from S3 storage with a single command.
2. Cross-Cloud Migration. Moving from AWS to Google Cloud or from one region to another becomes a trivial task. Velero packages resources and deploys them in the new location.
3. Environment Cloning. Need to quickly create an exact copy of production for testing? You make a prod backup, restore it to namespace staging. The database data will be current, not from a month-old dump.

How It Works Under the Hood

The project architecture is quite transparent. There's a server component that runs inside the cluster as an operator, and a CLI client for managing the process.

When you trigger a backup, here's what happens:

1. The client sends a request to the Velero API.
2. The Backup controller finds all the objects you specified (you can filter by labels or namespaces).
3. Velero makes requests to the Kubernetes API to collect JSON descriptions of resources.
4. In parallel, plugins for disk management are invoked (e.g., EBS in AWS or CSI drivers) to create data snapshots.
5. All of this is archived and sent to object storage (S3-compatible).

By the way, Velero supports Restic and Kopia. This means you can do incremental filesystem backups even where cloud snapshots aren't available.

Practical Example

Let's say we need to back up an entire application in namespace app-production. The terminal command would look something like this:

velero backup create production-backup --include-namespaces app-production

And if someone accidentally deletes that namespace a week later, recovery takes just a few minutes:

velero restore create --from-backup production-backup

Interestingly, Velero can modify resource parameters on the fly during restoration. For example, you can change the StorageClass for disks if the old class isn't available in the new cluster.

Compatibility Nuances

Velero's developers maintain a fairly strict compatibility matrix. Currently, version 1.18 is tested against the latest Kubernetes releases (up to 1.35). This is important because the Kubernetes API changes rapidly, and older tools often break when the cluster is upgraded.

The project is under the umbrella of the Cloud Native Computing Foundation (CNCF), which provides certain guarantees: it won't disappear tomorrow, and its security is being monitored.

Is It Worth Implementing

If you're running Kubernetes in production and have any data on disks (databases, queues, config stores), Velero is a must-have.

Who will definitely benefit from it:

• SRE engineers who need to sleep peacefully at night.
• Teams that frequently migrate between clusters.
• Developers who need fresh production data for debugging complex bugs.

Start with the official documentation at velero.io. It's quite detailed, though it may seem overwhelming at times. The main thing to remember is that a backup only exists when you've successfully restored from it at least once. Try doing this in a test cluster before trusting the tool with real data.