S3-Compatible Storage

github-backup can sync the JSON metadata of a backup (issues, pull requests, releases, labels, ...) and, optionally, release assets to any S3-compatible object store after the local backup completes.

Repository clones are not uploaded. The bare git mirrors, wikis and gist clones under <output>/<owner>/git/ stay local. For an offsite copy of the code itself use --mirror-to (see Mirroring) or a tool that copies directories (rclone sync, restic, borg).

If any upload or deletion fails, the run is recorded as incomplete: it exits with status 3 (see exit codes), the summary, --report, metrics and webhook ("partial") all list the failure, and the log says which objects failed and why. A failed S3 sync never undoes the local backup, which is already complete at that point.

Supported Providers

The endpoint must include its scheme (https://...). A bare host name is assumed to be https:// and logged as a warning.

Custom endpoints use path-style addressing (https://host/bucket/key). On AWS the bucket is addressed in the host name; a bucket name containing dots falls back to path-style on the regional endpoint.

Verification status. The S3 client is tested against an in-process fake server that verifies AWS Signature Version 4 independently and reproduces the four signature examples published by AWS. No real AWS, MinIO, B2, R2, Wasabi or Spaces server was available when this was written, so provider-specific behaviour is not verified by the project's tests. If a provider rejects requests, the error message carries the server's code and message; please report it.

Basic Usage

The bucket must already exist; github-backup never creates buckets.

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

github-backup octocat --output /backup --all \
  --s3-bucket my-backup-bucket \
  --s3-region us-east-1

Prefer the environment variables over --s3-access-key / --s3-secret-key: command-line values are visible to every user of the machine in the process list. Temporary credentials work too: set AWS_SESSION_TOKEN (or --s3-session-token) next to the key pair; it is signed as x-amz-security-token. AWS_PROFILE, ~/.aws/credentials, instance metadata and web-identity credentials are not read.

If --s3-bucket is set but a key is missing, the run reports exactly which credential is missing.

Provider-Specific Examples

Backblaze B2

export AWS_ACCESS_KEY_ID="$B2_KEY_ID" AWS_SECRET_ACCESS_KEY="$B2_APP_KEY"
github-backup octocat --output /backup --all \
  --s3-bucket my-b2-bucket \
  --s3-region us-west-004 \
  --s3-endpoint "https://s3.us-west-004.backblazeb2.com"

Cloudflare R2

export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$R2_SECRET_KEY"
github-backup octocat --output /backup --all \
  --s3-bucket my-r2-bucket \
  --s3-region auto \
  --s3-endpoint "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"

MinIO (self-hosted)

Create the bucket first (mc mb local/github-backup).

export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin
github-backup octocat --output /backup --all \
  --s3-bucket github-backup \
  --s3-region us-east-1 \
  --s3-endpoint "http://minio.internal:9000"

A plain http:// endpoint sends object names and data unencrypted; a warning is logged unless it is a loopback address.

Private certificate authority

The TLS client trusts the system certificate store. For a self-signed MinIO set SSL_CERT_FILE=/path/to/ca.pem (or SSL_CERT_DIR) for the process.

Proxies

HTTPS_PROXY and NO_PROXY apply to GitHub API calls only. S3 traffic connects to the endpoint directly; behind a proxy that blocks it, the upload phase fails with "cannot reach the endpoint".

Object Layout and Key Prefix

<prefix>/<owner>/json/<path relative to the json directory>

For example, --s3-prefix backups/github/ and owner octocat give backups/github/octocat/json/repos/hello/issues.json. Because the owner is part of every key, several owners can share one bucket and prefix without overwriting each other. The prefix may be empty and a trailing slash is optional.

Including Release Assets

By default only JSON metadata is synced. Add --s3-include-assets to also upload release assets (.../release_assets/<tag>/<file>):

--s3-bucket my-bucket --s3-include-assets

Warning: Binary assets can be large; each file is read into memory and uploaded (in 16 MiB parts above that size). Use a lifecycle rule to manage costs. Dropping --s3-include-assets later does not delete assets that are already in the bucket, not even with --s3-delete-stale.

Incremental Sync

Every uploaded object carries the SHA-256 of the local file as user metadata (x-amz-meta-sha256). On the next run the client asks the server for the object (HEAD) and skips the upload only when that digest and the size match. A file edited without changing its size is uploaded again. An object that has no digest (uploaded by another tool or an older version) is uploaded again once.

For encrypted uploads the stored value is a keyed digest (HMAC-SHA256 under a key derived from the encryption key), so the bucket never learns an unkeyed hash of your plaintext, and a new key makes every object upload again. See the Encryption guide.

If the credentials may not HEAD objects (no s3:GetObject, or no s3:ListBucket so a missing key reads as 403), the client cannot detect unchanged files and uploads everything on every run; a warning says so.

Deleting Stale Objects (--s3-delete-stale)

--s3-delete-stale removes objects under <prefix>/<owner>/json/ whose local file no longer exists. Safety rules:

• The listing covers <prefix>/<owner>/json/ only (with the trailing slash), so neighbouring prefixes such as github-backup-old/ are never touched.
• Nothing is deleted when the backup run had failures, when any part of the local tree could not be read, when the local tree holds no files, when any upload failed in the same run, or when the sync was aborted. The reason is logged.
• Release assets that were only left out of this run (no --s3-include-assets) are never considered stale.
• --dry-run never reaches this stage: a dry run skips the S3 sync altogether (see Dry Run).
• After you switch encryption on, the old plaintext objects are removed by --s3-delete-stale (they would otherwise stay readable in the bucket).

Deletion is permanent. On a bucket where you use this flag, enable bucket versioning (with a lifecycle rule that expires old versions): the tool deletes with a plain DeleteObject, so a wrongful delete leaves a recoverable version behind a delete marker.

Dry Run

--dry-run skips the S3 sync completely: the bucket is not contacted and nothing is uploaded or deleted. The sync code has a read-only preview mode (HEAD and LIST requests only), but the command line does not use it, so there is currently no way to preview an upload. To check credentials and bucket access before a real run, upload a small test with a throw-away prefix (--s3-prefix test/) and delete it afterwards.

Required Permissions

Minimum IAM policy (replace the bucket and prefix):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-backup-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["github-backup/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::my-backup-bucket/github-backup/*"
    }
  ]
}

• s3:ListBucket (on the bucket) is needed for the stale-object listing and, importantly, so that HEAD of a not-yet-uploaded key answers 404 instead of 403.
• s3:GetObject is what authorises HEAD; there is no s3:HeadObject action.
• s3:DeleteObject is only used with --s3-delete-stale; the multipart actions only for objects above 16 MiB.

Errors

A failed run reports the HTTP status, the S3 error code and message, and a hint:

Transient problems (5xx, SlowDown, dropped connections, stalls) are tried up to 5 times in total (the first attempt and four retries) with exponential back-off and jitter. A request is only abandoned when no data moved for 60 seconds, so a slow upload that keeps progressing is not cut off. A configuration-class error (bad credentials, missing bucket) stops the run after the first failure instead of repeating it for every file. Incomplete multipart uploads are aborted on any failure.

At-Rest Encryption

export BACKUP_ENCRYPT_KEY=$(openssl rand -hex 32)
github-backup octocat --output /backup --all --s3-bucket my-bucket

Encrypted objects receive a .enc suffix. See the Encryption guide for the wire format, what is and is not protected, key rotation and decryption.

S3 Flags Reference

The config file can set the bucket, region, prefix, endpoint, keys and s3_include_assets; --s3-delete-stale, the session token and the encryption key come from the command line or the environment.

Not Covered

--doctor / --check do not test S3 access, and --dry-run does not either. The TUI has no S3 settings: S3 sync is available from the command line and the config file only. Repository clones, wikis and gists are never uploaded; to copy them off-site use --mirror-to, or copy <output>/<owner>/git/ with a tool such as rclone sync, restic or borg.

Implementation Details

The S3 client uses a pure-Rust SigV4 implementation (no AWS SDK, no openssl). Signatures are HMAC-SHA256 computed with the sha2 and hmac crates; every request signs exactly the headers it sends. At-rest encryption uses aes-gcm from the same RustCrypto family.