Config File Reference

Configuration File Reference

Godwit Sync supports YAML configuration files as an alternative to CLI flags. Use --config (-f) to load a config file.

Basic Usage

godwit sync --config my-sync.yml
godwit sync -f my-sync.yml

# Override a config value with a CLI flag
godwit sync -f my-sync.yml --parallel 8

CLI Precedence

CLI flags always take precedence over config file values. If a flag is set on the command line, the corresponding config value is ignored.

Environment Variables

Config file values support environment variable expansion. Use $VAR or ${VAR} syntax to reference environment variables. Variables are resolved when the config file is loaded.

source:
  endpoint: minio.example.com:9000
  access_key: $AWS_ACCESS_KEY_ID
  secret_key: ${AWS_SECRET_ACCESS_KEY}
  session_token: ${AWS_SESSION_TOKEN}

Unset environment variables expand to an empty string.

Config File Structure

The configuration file uses YAML format with these top-level sections:

Source

Configure the source endpoint - local path or S3 bucket.

YAML Key	Type	CLI Flag	Description
source.url	string	--source	Source path or S3 URI (s3://bucket/prefix)
source.endpoint	string	--source-endpoint	S3 API endpoint (host:port)
source.auth_mode	string	--source-auth	Source credential mode (static, env, profile, iam, assume-role, web-identity, anonymous, auto)
source.access_key	string	--source-access-key	S3 access key. Required when auth_mode is static; optional for assume-role (omit to resolve caller from the default chain). Ignored for env, profile, iam, web-identity, anonymous, auto.
source.secret_key	string	--source-secret-key	S3 secret key. Paired with access_key: set both or neither.
source.session_token	string	--source-session-token	Optional session token for temporary credentials
source.profile	string	--source-profile	AWS profile name (used with auth_mode: profile)
source.role_arn	string	--source-role-arn	IAM role ARN for STS AssumeRole or AssumeRoleWithWebIdentity
source.role_session	string	--source-role-session	Session name for STS AssumeRole (CloudTrail). Honored by assume-role; web-identity uses an auto-generated session name.
source.token_file	string	--source-token-file	Path to a web-identity / OIDC JWT file on disk (used with auth_mode: web-identity). Falls back to AWS_WEB_IDENTITY_TOKEN_FILE. For EKS IRSA, Kubernetes projected tokens, IAM Roles Anywhere. It is not for CI platforms that expose the JWT via HTTP (use the platform's credential action plus auth_mode: env).
source.region	string	--source-region	S3 region
source.secure	bool	--source-secure	Use TLS for S3 connections
source.disable_keepalive	bool	--source-disable-keepalive	Disable HTTP keep-alive

Destination

Configure the destination endpoint - local path or S3 bucket.

YAML Key	Type	CLI Flag	Description
destination.url	string	--destination	Destination path or S3 URI
destination.endpoint	string	--destination-endpoint	S3 API endpoint (host:port)
destination.auth_mode	string	--destination-auth	Destination credential mode (static, env, profile, iam, assume-role, web-identity, anonymous, auto)
destination.access_key	string	--destination-access-key	S3 access key. Required when auth_mode is static; optional for assume-role (omit to resolve caller from the default chain). Ignored for env, profile, iam, web-identity, anonymous, auto.
destination.secret_key	string	--destination-secret-key	S3 secret key. Paired with access_key: set both or neither.
destination.session_token	string	--destination-session-token	Optional session token for temporary credentials
destination.profile	string	--destination-profile	AWS profile name (used with auth_mode: profile)
destination.role_arn	string	--destination-role-arn	IAM role ARN for STS AssumeRole or AssumeRoleWithWebIdentity
destination.role_session	string	--destination-role-session	Session name for STS AssumeRole (CloudTrail). Honored by assume-role; web-identity uses an auto-generated session name.
destination.token_file	string	--destination-token-file	Path to a web-identity / OIDC JWT file on disk (used with auth_mode: web-identity). Falls back to AWS_WEB_IDENTITY_TOKEN_FILE.
destination.region	string	--destination-region	S3 region
destination.secure	bool	--destination-secure	Use TLS for S3 connections
destination.disable_keepalive	bool	--destination-disable-keepalive	Disable HTTP keep-alive

Policy

Comparison and skip policies for object filtering.

YAML Key	Type	CLI Flag	Description
policy.compare	string	--compare-policy	Comparison policy (size,etag or size,mtime)
policy.skip	string[]	--skip	List of suffixes or patterns to skip
policy.match	string	--match	Regexp used to include only matching source object keys

Options

Transfer behavior settings - parallelism, retries, overrides.

YAML Key	Type	CLI Flag	Description
options.parallel	int	--parallel	Number of parallel sync workers
options.plan_parallel	int	--plan-parallel	Number of parallel planning workers
options.buffer_multiplier	int	--buffer-multiplier	Multiplier for the copy queue length
options.multipart_threshold	int64	--multipart-threshold	Multipart upload threshold in bytes
options.override	bool	--override	Overwrite existing destination objects
options.skip_tags	bool	--skip-tags	Skip reading object tags from source
options.unsupported_key_action	string	--unsupported-key-action	Action for unsupported S3 object keys: fail, skip, or warn
options.retry	int	--retry	Maximum retries per object
options.retry_backoff	string	--retry-backoff	Base delay between retries
options.ram_conflict_check	bool	--ram-conflict-check	Use in-memory map for case-conflict detection (faster but unbounded RAM)
options.incremental	bool	--incremental	Plan the delta against the prior completed run using only the local database (no destination calls)

Run

Run tracking - run ID, state driver and state file path.

YAML Key	Type	CLI Flag	Description
run.run_id	string	--run-id	Run identifier for tracking
run.state_driver	string	--state-driver	State backend driver
run.state_path	string	--state-path	Path to state database file
run.logs_dir	string	--logs-dir	Directory for observability logs

Output

Console output mode - UI, brief, or silent.

YAML Key	Type	CLI Flag	Description
output.ui	bool	--ui	Enable verbose console logging
output.brief	bool	--brief	Show only essential progress messages
output.silent	bool	--silent	Suppress all console output

License

License file path or inline license string.

YAML Key	Type	CLI Flag	Description
license.file	string	--license-file	Path to license file
license.inline	string	--license	Inline base64 license string

Rate Limit

Throttling controls - RPS, bandwidth, and concurrency.

YAML Key	Type	CLI Flag	Description
rate_limit.rps	float64	--rps	Requests per second limit (0 = disabled)
rate_limit.read_bps	int64	--read-bps	Read bytes per second limit (0 = disabled)
rate_limit.max_inflight	int	--max-inflight	Max concurrent uploads (0 = disabled)

Status

HTTP server address for metrics and status endpoint.

YAML Key	Type	CLI Flag	Description
status.addr	string	--status-addr	HTTP address for status/metrics server
status.drain_timeout	int	--drain-timeout	Seconds to keep the status/metrics server alive after the operation completes, allowing readers to scrape final data (default: 15)

Versioning

Version listing and source-key matching settings.

YAML Key	Type	CLI Flag	Description
versioning.mode	string	--version-mode	Version listing mode (latest, all, since:<RFC3339>, asof:<RFC3339\|duration>)

Object Lock

Object Lock replication settings for retention and legal hold.

YAML Key	Type	CLI Flag	Description
object_lock.enabled	bool	--object-lock	Replicate Object Lock retention and legal hold from source to destination

Sample Configurations

Local Filesystem → S3

Upload a local directory to an S3-compatible bucket.

source:
  url: "./data"

destination:
  url: "s3://my-bucket/backup"
  endpoint: localhost:9001
  access_key: access_key
  secret_key: secret_key
  secure: false

options:
  parallel: 3

output:
  ui: true

status:
  addr: ":8080"

S3 → Local Filesystem

Download objects from an S3 bucket to a local directory.

source:
  url: "s3://my-bucket/data"
  endpoint: localhost:9001
  access_key: access_key
  secret_key: secret_key
  secure: false

destination:
  url: "./downloads"

policy:
  skip:
    - .md5

options:
  parallel: 6

output:
  ui: true

S3 → S3

Copy data between two S3-compatible endpoints.

source:
  url: "s3://source-bucket/data"
  endpoint: localhost:9001
  access_key: access_key
  secret_key: secret_key
  secure: false

destination:
  url: "s3://dest-bucket/backup"
  endpoint: localhost:9101
  access_key: access_key
  secret_key: secret_key
  secure: false

policy:
  skip:
    - .md5

options:
  parallel: 9

output:
  ui: true

status:
  addr: ":8080"

Full Example (All Sections)

A complete configuration file showing every available option.

source:
  url: "s3://source-bucket/prefix"
  endpoint: source.storage.example.com
  access_key: SOURCE_ACCESS_KEY
  secret_key: SOURCE_SECRET_KEY
  region: us-east-1
  secure: true

destination:
  url: "s3://dest-bucket/backup"
  endpoint: localhost:9000
  access_key: access_key
  secret_key: secret_key
  secure: false

policy:
  compare: "size,etag"
  skip:
    - .md5
    - .tmp
  match: "^data/.*\.txt$"

options:
  parallel: 4
  plan_parallel: 8
  buffer_multiplier: 4
  multipart_threshold: 67108864
  override: false
  skip_tags: false
  unsupported_key_action: "skip"
  retry: 3
  retry_backoff: "1s"

run:
  run_id: "daily-backup"
  state_driver: sqlite
  state_path: "./godwit-sync.state.db"
  logs_dir: "./logs"

output:
  ui: true
  brief: false
  silent: false

license:
  file: "./license.json"

rate_limit:
  rps: 50
  read_bps: 104857600
  max_inflight: 10

versioning:
  mode: "all"

object_lock:
  enabled: true

status:
  addr: ":8080"

CLI-Only Flags

These flags have no config file equivalent and must be passed on the command line:

Flag	Description
--config (-f)	Selects which config file to load
--plan-only	Plan transfer without copying data
--resume	Resume a previously planned transfer
--register	Register a license to disk

CLI Reference

See the complete list of all CLI flags with types, defaults, and descriptions.

View CLI Reference

← Guides CLI Reference →