The 3-2-1 Backup Rule with S3-Compatible Storage
The 3-2-1 backup rule explained: keep 3 copies on 2 media with 1 offsite. Use S3-compatible storage and Godwit Sync for a verifiable offsite copy.
The 3-2-1 backup rule says to keep three copies of your data on two different kinds of storage, with one of those copies somewhere else. Peter Krogh wrote it down for photographers in 2009, and US-CERT adopted it in its Data Backup Options paper. The rule covers only the number and placement of copies. Copy frequency, retention, failure detection, and the recovery procedure are left to the implementation.
The layout most home labs and small teams run:
- local filesystem holds the working data
- second disk or host holds the local copy
- S3-compatible bucket in another location holds the offsite copy.
Godwit Sync handles the offsite leg. If your working data already lives in a self-hosted S3 service, the same design applies.
The 3-2-1 Rule: Three Copies, Two Media, One Offsite
The rule is three requirements. Each one covers a failure the other two do not.
Three copies is the working data plus two backups. Two, because the first backup can already be corrupted on the day you need it. A snapshot on the primary array does not count as a copy, because it lives on the same disks.
Two media puts the copies on different disks or storage systems. A backup on a second volume of the same array shares that array's controller, firmware, and power supply, so at least one backup has to live on hardware that a failure of the primary host cannot reach. Different failure modes are the point, not different technologies: a separate NAS, a second host, or an S3 bucket.
One offsite keeps at least one copy outside the building. It is the only copy that survives a site-level loss: fire, flood, theft, or ransomware that reaches every share on the LAN.
The offsite copy also needs to be separate administratively, not only physically. Two buckets in the same provider account do not give you that. Put the offsite bucket in its own account or tenant.
There is a 3-2-1-1-0 variant. It adds two requirements: one copy is immutable or offline, and backups verify with zero errors. On S3-compatible storage, bucket versioning together with Object Lock covers the immutable copy. Checksum verification catches objects that no longer match what was transferred, and a restore test confirms that the recovered data is usable.
The other common variants add copies and locations. The 3-2-2 variant keeps three copies on two media but calls for two offsite copies instead of one, so a provider outage or a lost account does not take the only copy outside the building. On S3-compatible storage that is a second bucket at a different provider, in its own account. The 4-3-2 variant goes further: four copies across three locations, two of them offsite. Both raise the odds that some copy survives, which matters when the data is irreplaceable. Neither adds a requirement for immutability or verification, so a locked and verified 3-2-1-1-0 layout stands up to ransomware and silent corruption better than an unlocked 4-3-2. Locking and verification take priority over additional copies.
Example Layout
| Copy | Location | Purpose |
|---|---|---|
| Primary | Application host, NAS, or local S3 service | Active data |
| Local backup | Separate disk, host, or storage appliance | Fast recovery from file, disk, or host failure |
| Offsite backup | S3-compatible bucket in another location | Recovery when the source site is unavailable |
The backup interval follows from the recovery point objective (RPO). With an RPO of 24 hours, at least one offsite run has to complete every day. The recovery time objective (RTO) works in the other direction: the volume you expect to restore, divided by the RTO, is the throughput the restore path has to sustain. That number decides whether a given provider and network link are adequate before you commit to them.
S3-Compatible Storage as the Offsite Copy
The rule leaves the offsite copy with three requirements: outside the building, under its own account, and able to be locked and verified. S3-compatible storage fits, and it keeps the location open: the object API is the same whether a provider hosts the bucket or you run the server software at a second site. Switching between them changes the endpoint and credentials, not the backup job. Godwit Sync writes each file as a plain object under its original path rather than packing files into a repository format, so any S3 client can list and download the backup, with or without Godwit Sync installed.
Not every S3-compatible service supports the same features behind that API. Before committing to a destination, run a test bucket through the full cycle and confirm that multipart uploads and checksum verification work against it. Check whether the provider supports versioning and Object Lock, how its lifecycle rules behave, and what its maximum object size is. When pricing the restore path, count request charges, minimum retention periods, retrieval fees, and egress, not only storage per gigabyte.
The credentials that write the backup deserve their own principal with only the bucket permissions the job needs. Keep them separate from the administrative credentials of the source system, and alert on authentication failures or policy changes that touch the backup bucket.
Make the First Offsite Copy
The source is a local directory or a mounted share. Run with --plan-only first. Godwit Sync scans the source, records the plan in its local state database, and prints the file count and total bytes without writing anything to the bucket:
godwit sync \
--source /mnt/tank/app-data \
--destination s3://app-data-offsite \
--destination-endpoint s3.REGION.example-cloud.com \
--destination-access-key <offsite-key-id> \
--destination-secret-key <offsite-secret> \
--compare-policy size,mtime \
--run-id offsite-321 \
--plan-only
The endpoint is a host name only, with no http:// or https:// prefix. TLS is on by default, so --destination-secure=false belongs only on a destination that deliberately serves plain HTTP. The example passes static access keys because that works everywhere, but it is not the safest option. When the destination supports it, prefer short-lived or automatically refreshed credentials through --destination-auth. AWS S3 Authentication Methods Compared walks through the choices. Whichever method you use, keep static keys out of checked-in scripts and shell history.
--compare-policy size,mtime compares filesystem entries by size and modification time, which are the signals a local file has to offer. Once the plan numbers look right, run the same command without --plan-only. When the run completes, Godwit Sync seals a baseline for offsite-321 in the state database, and every later run builds on it.
Run Incremental Copies
Every run after the first add --incremental and keeps the same --run-id. Godwit Sync reads the last sealed baseline, compares each source file against it, and copies only the files that are new or changed. Unchanged files are skipped without touching the bucket. For this to work, the state database has to survive between runs. Change the run ID, and Godwit Sync starts a separate history with no baseline to compare against.
Godwit Sync provides godwit schedule install which registers the run with the scheduler the host has:
- systemd timer or cron on Linux
- launchd on macOS
- Task Scheduler on Windows.
A scheduled job runs from a config file like the following:
source:
url: "/mnt/tank/app-data"
destination:
url: "s3://app-data-offsite"
endpoint: "s3.REGION.example-cloud.com"
access_key: "<offsite-key-id>"
secret_key: "<offsite-secret>"
policy:
compare: "size,mtime"
options:
incremental: true
run:
run_id: "offsite-321"
hooks:
on_success: "curl -fsS https://monitor.example.net/ping/offsite-321"
godwit schedule install \
--name offsite-321 \
--config /etc/godwit/offsite-321.yaml \
--daily-at 02:00
Verify the Stored Objects
godwit plan verify reads the destination objects that belong to a run and compares each one against the checksum recorded when it was transferred:
godwit plan verify \
--run-id offsite-321 \
--destination s3://app-data-offsite \
--destination-endpoint s3.REGION.example-cloud.com \
--destination-access-key <offsite-key-id> \
--destination-secret-key <offsite-secret>
The check confirms that what landed in the bucket is still what was sent. How to Verify S3 Migrations with Godwit Sync covers the rest of the verification workflow: listing runs and failed objects with godwit plan list, reading run status and object counts with godwit plan inspect --json, and a validation checklist to work through after the copy.
Immutability Is a Bucket Setting
With versioning enabled, an update writes a new object version instead of replacing the old one, and Object Lock can hold individual versions under GOVERNANCE or COMPLIANCE retention. Both are settings on the destination bucket. Godwit Sync writes into a locked bucket the same way it writes into any other.
Object Lock is the important control against ransomware. An attacker holding the backup credentials can delete or overwrite unlocked objects, and an incremental run cannot tell an encrypted file from a legitimately changed one, so it uploads the damage as a new version. With Object Lock in force, the earlier versions stay readable until their retention expires.
Before pointing production backups at a locked bucket, confirm how the provider implements Object Lock:
- which retention modes it supports
- what default retention applies
- and which permissions the backup principal needs.
Noncurrent versions stay billable until both retention and lifecycle rules allow their removal, so budget for them.
Test a Restore
A restore is a sync in the opposite direction: the bucket becomes the source and a fresh local directory becomes the destination. Give it its own run ID so the restore state stays apart from the backup history, and preview it before writing anything:
godwit sync \
--source s3://app-data-offsite \
--source-endpoint s3.REGION.example-cloud.com \
--source-access-key <offsite-key-id> \
--source-secret-key <offsite-secret> \
--destination ./restore-test \
--run-id offsite-321-restore-test \
--plan-only
Drop --plan-only to run it. Time the restore and validate the files with the software that consumes them. If the restore takes longer than the RTO allows, fix the procedure, the network path, or the storage tier now, before anything depends on it.