Changelog

New features

• Versioned sync now dedups against the destination's full version history. Re-running --version-mode all on a partly-synced key previously re-uploaded every version the destination happened to be missing under that exact version-id, even when the content was already present on the destination under a different version. The planner now walks the destination's full version list per key (newest-first) and finds an anchor: the newest source version whose content matches any destination version. Versions older than the anchor are skipped; versions newer are queued for upload. A run that was interrupted partway through, or a destination that was seeded from a different source, now incrementally uploads only the missing tail instead of the whole history.

Fixes

• ETag/MD5 false matches on versioned objects. A short-circuit in the ETag comparator returned true for any single-part ETag, so unrelated versions with the same size were silently treated as identical and skipped from sync. The comparator now compares MD5 on both sides directly. Affected versioned runs where the source and destination disagreed on a single-part object of matching size; re-run with the new release to repair.
• Null-version checksum sidecar written to the wrong path. On a versioned run, a NullVersionID task was writing its checksum to a .v.null.md5 sidecar instead of the plain un-versioned sidecar. Real version IDs continue to use the .v.<vid>.md5 form; only the null version writes the plain sidecar.

New features

• Web identity / OIDC auth mode. A new web-identity auth mode exchanges a short-lived JWT for temporary STS credentials via AssumeRoleWithWebIdentity. Covers IRSA (the projected service account token in EKS) and GitHub Actions (ACTIONS_ID_TOKEN_REQUEST_URL). New --source-token-file and --destination-token-file flags point at the JWT path; if omitted, AWS_WEB_IDENTITY_TOKEN_FILE is used. The token is re-read on every credential refresh, so rotating tokens work without restarting the process.
• assume-role without long-lived keys. When --access-key and --secret-key are omitted, assume-role now resolves caller credentials from the default chain (env → profile → IMDS) and exchanges them for a session bound to the role ARN. This is the standard pattern when an EC2/ECS host or EKS pod runs as one role and needs to assume another.
• plan-verify gains --destination-token-file. Matches the new web-identity flag on the sync command.

New features

• Full AWS credential chain support. S3 endpoints now support seven authentication modes via the new --source-auth and --destination-auth flags (and matching config fields): static (access key + secret), env (AWS environment variables), profile (~/.aws/credentials), iam (EC2/ECS instance role), assume-role (STS AssumeRole with configurable role ARN and session name), anonymous, and auto (chained fallback: static → env → profile → IAM). Previously only static credentials were supported.
• STS endpoint auto-resolution for assume-role. Regional STS endpoints are derived automatically from the S3 endpoint, so assume-role works out of the box against AWS without extra configuration.
• New auth flags on plan-verify. The plan-verify command gains --destination-auth, --destination-profile, --destination-role-arn, and --destination-role-session to match the sync command.

New features

• Environment variable expansion in YAML config. Config files now support $VAR and ${VAR} references that are expanded from environment variables before parsing. Useful for injecting credentials or environment-specific values without committing them to config files.
• RPS limiting for S3 listing. The --target-rps setting now also throttles ListObjects calls, preventing request-rate spikes during the planning phase.

Fixes

• Unbounded memory on case-colliding keys. Buckets with object keys that differ only by case could cause unbounded map growth during planning. Case conflicts are now detected via the state database by default, keeping memory bounded. Use --ram-conflict-check to opt into the faster in-memory approach when bucket size is manageable.
• Failed runs are now finalized correctly. Runs that hit early errors (Object Lock checks, plan failures) are finalized as failed instead of left dangling. Failed object counts and transferred bytes are accurately reported in run metrics.
• Shared read-rate limiter. A single ReadBPS rate limiter is shared across all workers instead of creating one per reader, giving more accurate bandwidth control.
• UI mode auto-scroll. The --ui progress list now scrolls to show the most recent items instead of always rendering the first N entries when the list exceeds terminal height.

New features

• Verify metrics in plan verify server. The /status endpoint now exposes verify counts alongside sync progress, giving dashboards and monitoring tools a single source of truth for both operations.
• Verify resume pre-seeding. When resuming a verify operation, previously completed counts are loaded from the task repository so progress starts from where it left off.
• --drain-timeout flag. Controls how long the process waits for in-flight operations to complete before shutting down. Also available as a config field.
• Full observability lifecycle for verify. The verify command now emits the same Prometheus metrics, status-server updates, and console events as the sync command.
• Live stats and object gauges during sync. Real-time object-level gauges are emitted during sync operations, improving visibility in Prometheus and Grafana dashboards.
• Pre-initialized Prometheus metric labels. Dashboard panels no longer show "No data" on startup; all metric labels are registered with zero values at initialization.

Fixes

• Tracker Reset semantics. Corrected the counter reset behavior and added pending counts to the progress tracker.
• License limit reached event. The brief console output now reports when the license object limit is reached, instead of silently stopping.

Fixes

• Exclude delete markers from verification. The verify command no longer attempts to verify S3 delete markers, which caused false failures when verifying versioned bucket migrations.

New features

• Version-aware S3 bucket migration. Godwit Sync can now migrate all object versions from a versioned S3 bucket, not just the latest version. Version IDs are tracked on both source and destination, and partial version history is reported during the operation.
• Object lock migration. Retention policies and legal holds are preserved when migrating objects between S3-compatible storage systems.
• --config flag for plan commands. Plan, sync, and verify commands can now read settings from a YAML configuration file instead of requiring all options as CLI flags.
• --unsupported-key-action=warn mode. Objects with keys that the destination does not support are logged with per-object events instead of failing the entire operation.
• --case-conflict and --unsupported filters. The plan list-objects command can now filter its output to show only objects with case-conflicting keys or unsupported key characters.
• Post-upload version tag replication. After uploading each version, a PutVersionTags call confirms that the destination has recorded the correct version ID, catching replication mismatches early.

Changes

• Match config moved to policy section. The match configuration block has moved from the versioning section to the policy section in the config file.

New features

• Config file support for verify. The verify command now accepts configuration from a YAML file, matching the config support already available in plan and sync.
• Read config from file. All commands that accept --config can now load settings from a file path.

Fixes

• Brief plan output shows zero counters. Fixed a bug where the non-UI console output displayed zero for all object and byte counters. The TTY message factory was reading flat payload keys instead of the nested objectStats struct.

New features

• --retry and --retry-backoff flags. Failed object transfers are now retried automatically. Configure the maximum number of attempts with --retry and the backoff interval with --retry-backoff.
• Sync direction in header. The console output header now shows the sync direction (e.g., s3→fs, fs→s3, s3→s3) for easier identification.
• New Prometheus counters. Additional metrics for monitoring transfer operations.

Fixes

• CGO enabled for SQLite. Previous builds were compiled with CGO_ENABLED=0, causing runtime crashes because go-sqlite3 requires CGO. All release binaries are now built with CGO enabled.
• ARM64 Docker builds stabilized. Fixed intermittent failures in ARM64 Docker builds running under QEMU emulation.
• SQLite atomic alignment warnings suppressed. Eliminated noisy compiler warnings on ARM targets.

Features

• S3-to-filesystem, filesystem-to-S3, and S3-to-S3 sync. Transfer objects between any combination of local filesystem and S3-compatible storage (AWS S3, MinIO, and other S3-compatible providers).
• Plan-based architecture. Every sync operation starts with a plan that inventories source and destination, calculates the diff, and records what needs to be transferred. Plans can be inspected before execution.
• Resume interrupted transfers. If a sync operation is interrupted, resume from where it left off without re-transferring completed objects.
• Verify after sync. A dedicated verify command compares source and destination after a sync to confirm all objects were transferred correctly.
• Interactive terminal UI. A Bubble Tea terminal interface shows real-time progress with object counts, byte counters, and transfer speed.
• Brief console mode. Run with --ui=false for a non-interactive log-style output suitable for scripts and CI pipelines.
• Prometheus metrics. Built-in metrics endpoint for monitoring sync operations with Prometheus and Grafana.
• License validation. Commercial license support via the Godwit license server.
• Multi-platform binaries. Pre-built binaries for Linux (amd64, arm64, armv6), macOS (amd64, arm64), Windows (amd64), and a Docker image.