Migrate, sync, and back up S3 storage with proof every run finished
Godwit Sync is a plan-first tool to migrate, sync, and back up S3-compatible object storage. Move only what changed between any S3 stores or from a local server, keep every version as a plain S3 object, and get proof every run finished with checksum verification and crash-safe resume.
Core Capabilities
Reliable S3 transfers, verified end to end
Every run is controlled, resumable, and verifiable.
Version History
Migrate all object versions
Rate Limited
Control read/write throughput
Resumable
Continue from exact checkpoint
Incremental Sync
Transfer only what changed, and prove the run finished
Plan Before You Move
Know exactly what will happen before any data moves. Generate migration plans, inspect them, modify as needed, and execute with confidence.
- Deterministic migration plan generation (--plan-only)
- Migration plans stored in SQLite
- Inspect, modify, and reuse plans
- Predictable execution every time
$ godwit sync \
--source s3://production-bucket \
--destination s3://backup-bucket \
--plan-only --brief
────────────────────────────────────────
PLAN CREATED
────────────────────────────────────────
✓ Run-ID: run-1770316589731184186
✓ Objects analyzed: 685,234
✓ Objects to transfer: 620,205
✓ Objects skipped: 65,029
✓ Data to transfer: 40.03 GB
✓ Data skipped: 4.7 GB
────────────────────────────────────────
End-to-End Checksums
Validate every object with real MD5 hashes, not just ETAGs.
Deterministic Runs
Identical inputs produce identical results, with no hidden randomness.
Parallel, Controlled Transfers
High throughput without sacrificing stability or safety.
Metadata Fidelity
Preserve object metadata exactly as stored at the source.
Chunk-Level Recovery
Recover from partial uploads without corrupting large files.
Object Lock Preservation
Replicate GOVERNANCE/COMPLIANCE retention and legal hold settings per version.
Production-Safe by Design
Built for environments where downtime and data loss are not options. Rate-limit operations, resume from any point, and keep your source systems healthy.
- Rate-limited reads and writes
- True stream-to-stream copy (no local staging)
- Resume from exact file or chunk
- Large file stability
- No overload of source systems
Every version, verified byte for byte
Back up local server directories or migrate S3 buckets without losing version history. Godwit Sync keeps every version, writes each change as a new S3 object version, verifies checksums, skips Glacier-class versions, and preserves Object Lock retention.
- Back up a local server directory to any S3-compatible bucket with incremental runs
- Migrate full or partial S3 bucket version history
- Store each change as a new S3 object version, never overwriting older versions
- Verify every version with .md5 sidecars and godwit plan verify
- Skip Glacier-class versions and preserve Object Lock retention and legal hold
# Plan a full version history migration
$ godwit sync \
--source s3://prod-bucket \
--destination s3://archive-bucket \
--version-mode all --plan-only
Planning...
⚠ Warning: 9 GLACIER objects detected.
Restore required before migration.
────────────────────────────────────────
PLAN CREATED
────────────────────────────────────────
✓ Run-ID: version-history-prod
✓ Objects analyzed: 23
✓ Objects to transfer: 14
✓ Objects skipped: 9
✓ Data to transfer: 48.20 GB
────────────────────────────────────────# Incremental sync transfers only what changed
$ godwit sync \
--incremental --brief --run-id daily-backup \
--source s3://prod \
--source-endpoint prod-rustfs:9000 \
--destination s3://backup \
--destination-endpoint backup-rustfs:9000
────────────────────────────────────────
PLAN READY
────────────────────────────────────────
✓ Run-ID: daily-backup
✓ Objects analyzed: 500,234
✓ Objects to transfer: 1,072
✓ Objects skipped: 499,162
✓ Data to transfer: 890 MB
✓ Data skipped: 1.4 TB
────────────────────────────────────────
RUN COMPLETED
────────────────────────────────────────
✓ Run-ID: daily-backup
✓ Objects transferred: 1,072
✓ Data transferred: 890 MB
✓ Total time: 47s
────────────────────────────────────────Sync Only What Changed
Run the same command from your own cron or systemd timer and forget it. The first execution is a full sync; every subsequent run compares each source object against confirmed-completed metadata and transfers only what changed. A 500,000-object bucket with 1,000 daily changes skips 499,000 destination lookups and transfers just the 1,000, then exits with a status endpoint and exit code that prove the run finished.
- Change detection by size+ETag content fingerprint (default): clock-skew-proof, with no false changes from mtime drift. Use size+mtime via --compare-policy for stores without reliable ETags
- Every transferred object gets a portable .md5 sidecar: a whole-object checksum independent of the provider's ETag (which differs for multipart uploads), so godwit plan verify can re-read and prove integrity on demand
- Baseline uses only confirmed-completed objects; crashed runs re-plan pending objects
- Proof every run finished: status REST endpoint, Prometheus metrics, automation-friendly exit codes
- Compatible with --plan-only: see the delta and estimate cost before it runs
Observable & Verifiable
Know exactly what's happening at every moment. Integrate with your existing monitoring stack and verify every byte after it lands.
- Verbose CLI UI with real-time progress
- REST status endpoint
- Prometheus-compatible metrics
- Predictable duration and progress estimates
- Built-in MD5 verify after every transfer
Data Movement, Done Right
Godwit Sync turns risky storage migrations into planned, observable, production-safe workflows for modern infrastructure teams.
Leaving MinIO? Move to S3, RustFS, or Ceph without losing the shape of your data.
Keep version history, Object Lock, checksum verification, and resumable transfers while moving away from a storage stack in maintenance mode.
Back up to S3 as plain objects.
Every version remains a normal object you can inspect and restore with familiar S3-compatible tools.