github-backup

github-backup is a GitHub backup tool written in Rust. It backs up the repositories, issues, pull requests, releases, gists, wikis and relationship data of a GitHub user or organisation to a local directory, optionally copies the JSON metadata to S3-compatible storage and the git data to another git host. TLS is rustls (no OpenSSL), the S3 client is a pure-Rust SigV4 implementation (no AWS SDK), and an optional full-screen TUI is available with --tui.

Documentation version. These pages describe the current development branch (main). The latest release, v0.3.2, predates many of the behaviours described here (exit status 3, per-repository incremental state, lossless JSON, --doctor, restore changes, and more); the Unreleased section of the changelog lists them. See Installation for how to get a matching build.

Feature Highlights

What It Does Not Back Up

• Discussions and Projects: GitHub's REST API has no endpoints for them (--discussions and --projects are accepted but do nothing).
• Actions secrets and variables, code-scanning and Dependabot alerts, rulesets, issue attachments, commit comments and statuses.
• Git LFS objects unless you pass --lfs (requires git-lfs).
• Repository clones are not uploaded to S3, only the JSON metadata.
• It cannot see what the token cannot see: other users' private data, or categories the token lacks permission for (a 403/404 is skipped with an INFO line and is not counted as a failure).
• Pull requests, comments and reactions cannot be restored; --restore handles labels, milestones and issues only.

See What Is Not Restored, or Not Backed Up.

Design Principles

• No OpenSSL, no reqwest, no AWS SDK: TLS via rustls, HTTP via hyper.
• No unsafe in the workspace's own source.
• Trait-based: BackupClient, Storage and GitRunner make the engine testable without network or filesystem.
• Credentials stay out of argument lists, URLs and files: git receives the token through an environment variable and a host-scoped credential helper.
• Rate-limit aware: rate-limit responses are waited out and retried.
• A backup that is incomplete says so: see Monitoring.

Quick Links

• Installation
• Quick Start
• Interactive TUI
• CLI Reference
• Config File (TOML)
• Docker
• Architecture
• GitHub Repository

Installation

github-backup is distributed in three ways:

1. Pre-built binary from the GitHub Releases page
2. Docker / Docker Compose from GHCR
3. Source install via cargo install --git

Not on crates.io. This project ships as an application, not a library. Every workspace crate is marked publish = false, so cargo install github-backup from the default registry will not work.

Which version does this documentation describe? It describes the current development branch (main). The latest published release, v0.3.2, predates a large set of changes; among others it has no --doctor, --check, --list-scopes, --print-config-template, --full, no exit status 3 and no per-repository incremental state; in v0.3.2 the second and later runs applied the previous run's timestamp automatically, which could overwrite issues.json with only the issues that had changed. The complete list is the Unreleased section of the changelog. Until the next release is published, use a source build from main (below) to get what these pages describe. github-backup --version prints the same number for a release and for a build from main before its version is bumped, so check the changelog if in doubt.

1. Pre-built binary

Every release publishes binaries for five targets, each with a .sha256 checksum file, plus a combined SHA256SUMS.txt:

(The static Linux binaries are built by the release workflow, which checks with file and ldd that they have no dynamic dependencies. The v0.3.2 Linux x86_64 binary, by contrast, was linked against glibc and needed glibc 2.39 or newer.)

# Linux x86_64 example: set VERSION to a version from the Releases page
VERSION=X.Y.Z
TARGET=linux-x86_64

curl -LO "https://github.com/tomtom215/github-backup-rust/releases/download/v${VERSION}/github-backup-${TARGET}"
curl -LO "https://github.com/tomtom215/github-backup-rust/releases/download/v${VERSION}/github-backup-${TARGET}.sha256"

# Verify the SHA-256
sha256sum -c "github-backup-${TARGET}.sha256"

# Install into /usr/local/bin
install -m 0755 "github-backup-${TARGET}" /usr/local/bin/github-backup

On macOS use shasum -a 256 -c instead of sha256sum -c.

Verify build provenance (optional)

Releases published after v0.3.2 (the first built by the current release workflow) attach a signed GitHub build-provenance attestation to each of the five binaries. With the GitHub CLI:

gh attestation verify "github-backup-${TARGET}" \
  --repo tomtom215/github-backup-rust

This proves that GitHub Actions built that file from this repository. It covers the five binaries only: not the .sha256 files, SHA256SUMS.txt or the container images (the images are not attested). There is no SBOM, and v0.3.2 and earlier releases have no attestation. The checksum files only protect against corrupted downloads; an attacker who can replace the binary can replace its checksum, which is what the attestation is for.

2. Docker / Docker Compose

Multi-architecture images (linux/amd64, linux/arm64) are published to GHCR for every release:

docker pull ghcr.io/tomtom215/github-backup-rust:latest   # latest stable release
docker pull ghcr.io/tomtom215/github-backup-rust:X.Y.Z    # a specific release
docker pull ghcr.io/tomtom215/github-backup-rust:X.Y      # latest patch of X.Y

(latest and X.Y are only moved by stable releases; a pre-release has just its own tag.) Note: :latest is the last published release, so until the next release it is v0.3.2 and lacks the entrypoint wrapper, the diagnostics and the other changes described in this documentation. Build the image from main (docker build -t github-backup .) to try them.

Ad-hoc run

docker run --rm \
  -e GITHUB_TOKEN \
  -v "$PWD/backups:/backup" \
  ghcr.io/tomtom215/github-backup-rust:latest \
  octocat --output /backup --all

-e GITHUB_TOKEN (no value) passes the variable from your shell without putting the token into the docker command line.

Docker Compose

The repository ships a docker-compose.yml with profiles for local backups, AWS S3, Backblaze B2, a MinIO or other S3-compatible server you run, and Codeberg / Forgejo / Gitea and GitLab mirroring, plus doctor, tui and verify. It reads secrets from a .env file next to it.

git clone https://github.com/tomtom215/github-backup-rust
cd github-backup-rust

cp compose.example.env .env
chmod 600 .env
$EDITOR .env            # set GITHUB_TOKEN and what the profile you use needs

# Local filesystem backup to ./backups/
docker compose run --rm backup octocat --all

# AWS S3 backup
docker compose --profile s3 run --rm backup-s3 octocat --all

# Codeberg mirror
docker compose --profile codeberg run --rm backup-codeberg octocat --all

The default backup service mounts one volume, ./backups (or BACKUP_DIR) at /backup. No config file is mounted; to use one, mount it for the run:

docker compose run --rm -v "$PWD/config.toml:/etc/github-backup/config.toml:ro" \
  backup --config /etc/github-backup/config.toml

The Docker guide has the details (user IDs, profiles, scheduling, Kubernetes).

3. Build from source

Requires a Rust toolchain meeting the MSRV in Cargo.toml (currently 1.88) and git at run time. Install via rustup.

# Track main (what this documentation describes)
cargo install --locked --git https://github.com/tomtom215/github-backup-rust \
  github-backup

# Or pin to a released tag (see the Releases page)
cargo install --locked --git https://github.com/tomtom215/github-backup-rust \
  --tag vX.Y.Z \
  github-backup

The binary lands in $CARGO_HOME/bin (by default ~/.cargo/bin), which a standard rustup install puts on your $PATH.

Or clone and build manually:

git clone https://github.com/tomtom215/github-backup-rust
cd github-backup-rust
cargo build --release --locked -p github-backup
sudo install -m 0755 target/release/github-backup /usr/local/bin/

(A glibc-linked build from source runs on the system it was built on. For a portable static binary, build with the Dockerfile's export stage: docker build --target export --output type=local,dest=out ..)

Verify installation

github-backup --version
github-backup --help
github-backup octocat --doctor      # checks git, the output directory, network and token

Shell completions

github-backup generates tab-completion scripts for all major shells via --completions <SHELL>. Run the one-time setup below, then open a new terminal (or source your shell's config file).

Bash

github-backup --completions bash >> ~/.bash_completion

Or write to /etc/bash_completion.d/github-backup (needs sudo) if your distribution loads completions from there.

Zsh

mkdir -p ~/.zfunc
github-backup --completions zsh > ~/.zfunc/_github-backup

Add these lines to ~/.zshrc once (before any compinit call):

fpath=(~/.zfunc $fpath)
autoload -Uz compinit && compinit

Then reload: exec zsh.

Fish

github-backup --completions fish > ~/.config/fish/completions/github-backup.fish

Fish loads files from ~/.config/fish/completions/ automatically.

PowerShell

github-backup --completions powershell >> $PROFILE

Reload your profile with . $PROFILE or start a new session.

Elvish

github-backup --completions elvish > ~/.config/elvish/lib/github-backup.elv

Then add use github-backup to ~/.config/elvish/rc.elv.

System requirements

Quick Start

1. Get a GitHub Token

Create a personal access token. For a complete backup of your own account a classic token with repo, gist, read:org and read:packages covers everything; for public data no token is needed. The scopes and fine-grained permissions per category are in Authentication.

Export it as an environment variable (not on the command line, where ps and your shell history can see it):

export GITHUB_TOKEN=ghp_your_token_here

2. Check the Setup

github-backup octocat --output /var/backup/github --doctor

--doctor checks that git is installed, that the output directory is writable, that the API is reachable and that GitHub accepts the token. It does not check disk space or scopes.

3. Run Your First Backup

Option A: Interactive TUI

github-backup octocat --tui

The TUI pre-fills the owner (and token and output directory, if given), shows all settings in tabbed panels and lets you start the backup and watch live progress. See the Interactive TUI guide.

Option B: Command line

Back up everything for a user:

github-backup octocat --output /var/backup/github --all

(--all includes private repositories and secret gists when the token belongs to octocat; see Private repositories.)

Back up only repositories and issues of an organisation:

github-backup my-org \
  --output /var/backup/github \
  --org \
  --repositories \
  --issues

The command prints a plan, then progress, then a summary. Check the exit status: 0 means complete, 3 means the backup finished but something could not be backed up (the summary lists it; run again to retry), 1 means it could not run at all. See Exit Codes.

4. Explore the Output

/var/backup/github/
├── .github-backup.lock
└── octocat/
    ├── git/
    │   ├── repos/
    │   │   ├── Hello-World.git/        ← mirror clone
    │   │   └── Spoon-Knife.git/
    │   ├── wikis/
    │   │   └── Hello-World.wiki.git/
    │   └── gists/
    │       └── abc123.git/
    └── json/
        ├── repos.json
        ├── starred.json
        ├── backup_state.json
        ├── backup_history.json
        └── repos/
            └── Hello-World/
                ├── info.json
                ├── issues.json
                ├── issue_comments/1.json
                ├── pulls.json
                ├── releases.json
                ├── labels.json
                └── milestones.json

(A selection; which files appear depends on the flags.) The full listing is on Output Directory Layout.

5. Run It Again

Run the same command again whenever you like (or schedule it with systemd or cron). Git clones are updated in place and only changed issues and pull requests are re-fetched; nothing already captured is lost. See Incremental runs.

6. Common Recipes

Selective backup with higher concurrency

github-backup octocat \
  --output /backup \
  --repositories --issues --pulls --releases \
  --concurrency 8

Shallow clone (saves disk space)

github-backup octocat \
  --output /backup \
  --repositories \
  --clone-type shallow:10

Dry run (preview, writes nothing)

github-backup octocat --output /backup --all --dry-run

A dry run lists the repositories it would back up and writes nothing: no files, no lock, no state, no report and no network call except the read-only listing. Owner-level data and gists are skipped.

Using a config file

mkdir -p /etc/github-backup
cat > /etc/github-backup/config.toml <<'EOF'
owner = "octocat"
output = "/var/backup/github"
concurrency = 8
repositories = true
issues = true
pulls = true
releases = true
wikis = true
EOF
chmod 600 /etc/github-backup/config.toml

github-backup --config /etc/github-backup/config.toml

(The token comes from GITHUB_TOKEN.)

Push mirror to Codeberg after the backup

export MIRROR_TOKEN=your_codeberg_token
github-backup octocat \
  --output /backup \
  --repositories \
  --mirror-to https://codeberg.org \
  --mirror-owner your_codeberg_username

S3 sync of the metadata

export AWS_ACCESS_KEY_ID=...  AWS_SECRET_ACCESS_KEY=...
github-backup octocat \
  --output /backup \
  --all \
  --s3-bucket my-backup-bucket \
  --s3-region us-east-1

The bucket must exist. Only the JSON metadata goes to S3, not the repository clones; see S3.

Next Steps

• Interactive TUI: full-screen interface with live progress
• Authentication: tokens, scopes and the device flow
• Backup categories: what each flag backs up, and what it cannot
• CLI Reference: all flags and the exit codes
• Monitoring: get told when a backup fails
• Docker: containers and scheduled backups

Authentication

github-backup supports three ways to authenticate: a personal access token (PAT), the GitHub OAuth device flow, and no credential (public data only).

Personal Access Token (Recommended)

The simplest and most reliable method for scheduled backups.

Creating a Token

Classic PAT

1. Open Settings → Developer settings → Personal access tokens → Tokens (classic).
2. Generate new token (classic).
3. Select the scopes from What each category needs. For a complete backup (--all, organisation or personal) select repo, gist, read:org and read:packages.
4. Copy the token.

Fine-grained PAT

1. Open Settings → Developer settings → Personal access tokens → Fine-grained tokens.
2. Choose the resource owner (your account, or the organisation whose repositories you back up; the organisation may need to approve the token) and repository access (All repositories or a selection).
3. Grant the repository permissions listed in the table below.
4. Fine-grained tokens cannot read some account-level resources (gists and packages are classic-only according to GitHub's documentation); use a classic token if you need --gists, --starred-gists or --packages.

Using the Token

Environment variable (preferred: the token stays out of shell history and the process list):

export GITHUB_TOKEN=ghp_dummy_token_for_illustration
github-backup octocat --output /backup --all

Other ways, in order of preference: a config file with mode 0600 (token = "..."; the tool warns if the file is readable by others) and the --token flag. A command-line value is visible in ps to every user of the machine and ends up in the shell history (the tool warns about that for --encrypt-key, not for --token), so avoid --token on shared hosts.

The token is only ever sent to the GitHub API host you configured, as Authorization: Bearer ..., and to git through a credential helper limited to the clone URL's host. It is not written to disk by the tool and is removed from error messages.

What Each Category Needs

The table maps each flag to the classic scope and the fine-grained permission GitHub documents for the endpoint it uses. It is derived from GitHub's documentation and has not been exercised against live accounts by the project: check with --doctor (token accepted) and a trial run, and look at the failures it reports.

Notes:

• A user's private repositories and secret gists are listed only when the token belongs to that user, see Private repositories.
• For organisation targets (--org) the token's owner must be able to see the repositories. If the organisation enforces SAML single sign-on, authorise the token for it (Settings → token → Configure SSO), otherwise GitHub answers 403.
• When a category is not permitted (HTTP 403 or 404) the tool skips that item with an INFO line instead of failing; see Repository Metadata. A repository that cannot be listed or cloned at all is a failure (exit status 3).
• github-backup --list-scopes <flags> prints suggested scopes for the flags you give, but it does not expand --all (it prints only public_repo repo) and it suggests user:follow and admin:public_key for --followers and --deploy-keys, which only read. Use the table above rather than that output for --all.

OAuth Device Flow

The device flow signs in interactively without creating a long-lived PAT; the token is not stored, so every run needs the login. It is unsuitable for unattended runs.

Prerequisites

1. Create an OAuth App:
   • Application name: anything (for example github-backup)
   • Homepage URL: any valid URL
   • Authorization callback URL: http://localhost (not used by the device flow)
   • tick Enable Device Flow in the app's settings
2. Copy the Client ID.

Running Device Flow

github-backup octocat \
  --device-auth \
  --oauth-client-id Iv1.xxxx \
  --oauth-scopes "repo gist read:org read:packages" \
  --output /backup \
  --all

You will see:

──────────────────────────────────────────────────────
  GitHub OAuth device authorisation
──────────────────────────────────────────────────────
  1. Open:  https://github.com/login/device
  2. Enter: ABCD-1234
──────────────────────────────────────────────────────
  Waiting for authorisation…

Open the URL, enter the code and authorise the app; github-backup polls for the token. If you do not authorise in time the flow ends with an "expired" error; just run the command again. --oauth-client-id is only valid together with --device-auth.

Scopes

The default --oauth-scopes is "repo gist read:org". It does not cover --packages (add read:packages). Narrow it if you back up fewer categories; see the table above.

Unauthenticated Access (Public Data Only)

Omit the token to back up public data:

github-backup octocat --output /backup --repositories --issues --releases

At startup the tool warns:

WARN no GitHub credential supplied — running unauthenticated. Limited to public data and 60 requests / hour. Set GITHUB_TOKEN, pass --token, or use --device-auth for a full backup.

Unauthenticated requests are limited to 60 per hour per IP address, which accounts with many repositories or issues exhaust quickly. If you name a flag that cannot work anonymously (--private, --hooks, --deploy-keys, --collaborators, --org-members, --org-teams, --actions, --action-runs, --packages, --discussions, --projects) the tool refuses to start (exit status 1). That check does not look inside --all, so do not use an anonymous --all.

Verify the Token

github-backup octocat --doctor

--doctor asks GitHub whether the token is accepted (GET /rate_limit, which costs no quota). It does not check scopes or permissions.

Security Best Practices

1. Pass the token in GITHUB_TOKEN (or a 0600 config file), not with --token.
2. Prefer a fine-grained token limited to what you need, or a classic token with only the scopes in the table above.
3. Rotate tokens regularly and revoke one immediately if it may have leaked.
4. chmod 600 /etc/github-backup/config.toml.
5. Never commit tokens to version control.

Interactive TUI

github-backup ships a full-screen terminal user interface built with Ratatui 0.30. Pass --tui and the tool enters the TUI instead of running non-interactively.

# Recommended first-run experience
export GITHUB_TOKEN=ghp_your_token_here
github-backup octocat --tui

# Flags pre-fill the Configure screen
github-backup octocat --tui --token "$GITHUB_TOKEN" --output /var/backup/github \
  --all --private --dry-run --org --concurrency 8 --manifest

The command line is resolved exactly as for a normal run, so --all, --private, --forks, --dry-run, --org, --full, --since, --include-repos, --clone-type, --concurrency, --manifest and the individual category flags all arrive in the Configure screen as initial values. Review and adjust them, then start the backup. Not carried over: --config files, --clone-host, and every option listed under What the TUI cannot do.

The TUI needs an interactive terminal on stdout. Without one (a pipe, cron, docker run without -t) it prints a one-line explanation and exits with status 2.

Screen Overview

The title bar lists five screens. Press the number key to switch (not while a backup is running, and not while editing a field). On narrow terminals the title bar shrinks to digits.

 github-backup v0.3.2  [1]Dashboard  [2]Configure  [3]Run  [4]Verify  [5]Results

Terminals smaller than 30x8 show a "Terminal too small" notice instead of a clipped screen.

1 — Dashboard

Owner, output directory, token status (never shown, only "configured") and the last recorded run: time, repository count and result (complete, or INCOMPLETE (N failures)). The last run is read from <output>/<owner>/json/backup_history.json, falling back to backup_state.json, and refreshed whenever you return to the Dashboard.

2 — Configure

Six tabs:

Full backup ignores the saved incremental state and re-fetches everything. It cannot be combined with Since, as on the command line. The Output tab ends with a note: Command line only: mirror push, S3 sync, JSON report, Prometheus metrics, webhook notification and device-flow sign-in.

The focused field's help text is shown under the list when there is room. Starting a backup validates the form (owner, token, https:// API URL, date format, concurrency range) and shows what to fix in a dialog.

3 — Run

Live view of an active backup:

• Progress gauge — repositories finished out of discovered, with a failure count. As soon as one repository fails the title reads INCOMPLETE: N failed and the bar turns red.
• Repo list — >> running, ok done, !! failed (with the reason), -- skipped. Failed repositories are pinned to the top so they never scroll away.
• Log panel — one row per entry (long lines are cut, never wrapped), so the newest entry is always visible while following. Only this tool's own INFO and above are shown by default; set RUST_LOG to change that.
• Counters — repos finished/total, failed, elapsed (stops when the run ends), and key hints.

With no backup running this screen just says so; Esc or q leave it.

4 — Verify

Checks the JSON files listed in backup_manifest.json against the disk. It does not cover git mirrors. The manifest exists only if you ran a backup with Write SHA-256 Manifest (or --manifest); otherwise Verify says so. v starts it.

5 — Results

The verdict follows the engine, never the repo counters alone:

Below the headline: counters (repositories discovered / backed up / skipped / failed, gists, issues, pull requests, workflows, discussions) and, when anything failed, a scrollable Failures list with the scope (usually owner/repo), the step (clone, wiki, issues, …) and the message. The selected failure's full text is shown underneath when there is room.

Key Reference

Global

Dashboard

Configure

While editing a text field: Enter saves, Esc discards the change, Backspace deletes. Pasting inserts a single clean line; a paste never triggers shortcuts. The Concurrency field accepts digits only. There is no cursor movement inside a field.

Run

Verify

Results

What the TUI cannot do

These exist only on the command line, and the TUI does not offer them:

• mirror push (Gitea/GitLab), S3 sync, JSON report, Prometheus metrics, webhook notification;
• device-flow sign-in (a token is required);
• --config files and --clone-host;
• the CLI-level output lock.

What the TUI does after a run, like the CLI: the engine writes the incremental state, the TUI appends a run-history entry (shown on the Dashboard, including the failure count) and, if enabled and not a dry run, writes the manifest. A manifest that cannot be written counts as a failure of the run.

Terminal safety

Raw mode, the alternate screen and bracketed paste are always restored on exit, on a panic (the message is printed after the terminal is restored), and on SIGINT, SIGTERM, SIGHUP and SIGQUIT. Such a signal cancels a running backup first (git is killed, the lock released) and exits with 128 + the signal number. A second signal exits immediately. SIGKILL cannot be handled; run reset if you ever use it. Closing the terminal window ends the process.

Architecture Notes

The TUI lives in the github-backup-tui crate and drives the same BackupEngine as the CLI. Per-repository progress and outcome arrive over the engine's event channel; the final verdict and failure list come from BackupStats. A tracing layer forwards log events to the Run screen's log panel. Terminal input is read on a dedicated thread; the async loop redraws only when something changed.

See Architecture for the crate dependency graph.

Backup Categories

github-backup organises backup targets into distinct categories. Each category can be enabled individually with a flag, or all can be enabled at once with --all.

Every JSON file a category writes holds GitHub's complete API response (all properties, GitHub's key order), not a summary: see Output Directory Layout.

Repositories

Clone Types Explained

How the clone types behave in detail:

• Updates are fetches into the existing directory; nothing is cloned twice. Branches and tags deleted on GitHub are kept in the clone unless --prune is given; force-pushed branches are overwritten either way (see Clones Follow GitHub).
• A fresh clone is made in a hidden staging directory next to the target and renamed into place only when it is complete, so an interrupted clone leaves nothing under the real name (the staging directory is removed on the next run).
• A shallow clone contains only the repository's default branch (and the tags that point into it) at first, because git's --depth implies a single branch for the initial clone; the next update fetches every branch at the requested depth.
• --lfs adds git lfs fetch --all origin after the mirror update (the clone itself is always a mirror: --lfs overrides --clone-type). It needs git-lfs on the PATH; without --lfs the clone holds LFS pointer files only, not the large objects.
• A git command that prints no output for 10 minutes is treated as stalled and killed; a clone that keeps making progress can take as long as it needs.
• --prefer-ssh clones over SSH; the token is then not used for git and git needs working SSH credentials.

Example:

# Mirror clone (default), recommended
github-backup octocat --output /backup --repositories

# Shallow clone, last 5 commits
github-backup octocat --output /backup --repositories --clone-type shallow:5

Output: <output>/<owner>/git/repos/<repo>.git/ (<repo>/ for --clone-type full)

Private repositories

GitHub's public listing (GET /users/<owner>/repos) never contains private repositories. When the token belongs to the account you are backing up (the login is compared case-insensitively), the tool lists GET /user/repos?affiliation=owner&visibility=all instead, which includes the account's own private repositories, and --private then takes effect. Only repositories the account owns are listed this way; repositories owned by organisations are backed up with --org.

For any other user, without a token, or with a token that cannot call GET /user (a GitHub App installation token; a warning is logged), only public repositories are visible. Organisation targets (--org) list private repositories the token can see, as before.

Issues

Issues and pull requests share one number space, and GitHub's issues API lists pull requests too, so --issue-comments and --issue-events also write issue_comments/<n>.json and issue_events/<n>.json for pull requests. The events come from /issues/<n>/events; the richer /timeline (cross-references, commits, reviews) is not fetched.

Output: <output>/<owner>/json/repos/<repo>/issues.json, issue_comments/<n>.json, issue_events/<n>.json

Pull Requests

Output: <output>/<owner>/json/repos/<repo>/pulls.json, pull_comments/<n>.json, pull_commits/<n>.json, pull_reviews/<n>.json

A pull request's conversation thread and events are written by --issue-comments / --issue-events (see Issues above).

Releases

Warning: --release-assets can consume significant disk space for projects with large binary releases.

Output: <output>/<owner>/json/repos/<repo>/releases.json, assets in release_assets/<tag>/<file> (+ <file>.sha256)

Wikis

Output: <output>/<owner>/git/wikis/<repo>.wiki.git/

Repository Metadata

Output: <output>/<owner>/json/repos/<repo>/labels.json, milestones.json, topics.json, branches.json, deploy_keys.json, collaborators.json, etc.

Note: --hooks, --deploy-keys, --collaborators, --security-advisories, the protection rules of --branches, --actions and --environments need admin or equivalent access. When GitHub answers 403 or 404 for a repository the tool treats it as "not available to this token", writes an INFO line and skips that file. This is not counted as a failure and does not change the exit status, so a token without the needed permission yields a backup without those files and exit status 0. Look for the file in the output (or for skipping in the log) when you rely on these categories.

Gists

Output:

• Git (owned gists only): <output>/<owner>/git/gists/<gist-id>.git/
• Metadata: <output>/<owner>/json/gists/<gist-id>.json, index.json
• Starred gists: <output>/<owner>/json/gists/<gist-id>.starred.json, starred_index.json

--starred-gists records the gist objects GitHub lists (description, owner, file names and sizes, URLs); it does not clone their contents.

GitHub's public listing (GET /users/<owner>/gists) omits secret gists. When the token belongs to the account being backed up, GET /gists is used and secret gists are included; otherwise only public gists are visible.

User / Organisation Data

Output: <output>/<owner>/json/starred.json, watched.json, org_members.json, org_teams.json, etc. Cloned starred repos: <output>/<owner>/git/starred/<upstream-owner>/<repo>.git

Note: --org-members and --org-teams are silently skipped for user targets. --clone-starred is intentionally omitted from --all due to its potentially large footprint.

GitHub Actions

--actions saves workflows.json to each repository's metadata directory. The actual workflow YAML files are already captured by the git clone; this flag records the API-level metadata that is not part of the repository tree (workflow IDs, states, badge URLs).

--action-runs writes one file per workflow (workflow_runs_<id>.json) with the workflow's runs; every page of GitHub's response is fetched, so this is the full retained history. This can be very large for active repositories; opt in deliberately. It is omitted from --all.

Output: <output>/<owner>/json/repos/<repo>/workflows.json, workflow_runs_<id>.json

Token scope: actions:read or a classic repo token is sufficient. Repositories with Actions disabled return 404, which is logged and skipped.

Deployment Environments

Environments model deployment targets such as staging or production. Their configurations include protection rules (required reviewers, wait timers) and branch policies that gate automated deployments. Backing up this metadata makes it possible to audit and reproduce deployment gate configurations without a live GitHub connection.

Output: <output>/<owner>/json/repos/<repo>/environments.json

Note: Repositories without environments return 404, which is logged and skipped silently.

Discussions

Not supported by the GitHub REST API. GitHub exposes Discussions through GraphQL only; the REST route this flag calls (GET /repos/<owner>/<repo>/discussions) does not exist in GitHub's published API description (github.com, GHEC, GHES 3.17-3.19) and answers 404. The flag is accepted, a single warning per run says that nothing is backed up, and no discussions.json is written. Do not rely on it for a backup of Discussions.

Classic Projects

Not supported by the GitHub REST API. GitHub sunset Classic Projects; the REST routes this flag calls (/repos/<owner>/<repo>/projects, /projects/<id>/columns) are absent from GitHub's published API description and answer 404/410. The flag is accepted, a single warning per run says that nothing is backed up, and no projects.json is written. Projects v2 are not implemented.

GitHub Packages

Iterates over the supported package ecosystems (container, npm, maven, rubygems, nuget, docker) and saves the package list and version metadata to the owner's JSON directory. Requires the read:packages OAuth scope.

Output: <output>/<owner>/json/packages_<type>.json

The --all Flag

--all enables every category above (including the non-functional --discussions and --projects, which log a warning) except:

• --lfs (requires git-lfs to be installed)
• --prefer-ssh (requires SSH keys to be set up)
• --prune (affects update behaviour)
• --action-runs (can be very large for active repositories)
• --clone-starred (can consume substantial disk space)
• --concurrency (set separately)

--forks and --private are part of --all. The excluded options can be given next to --all and are honoured (--all --clone-starred, --all --action-runs, --all --lfs). Individual category flags cannot be combined with --all on the command line.

github-backup octocat --token $GITHUB_TOKEN --output /backup --all

Issues & Pull Requests

Issues

Enable issue backup with one or more of:

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --repositories \
  --issues \
  --issue-comments \
  --issue-events

What is backed up

--issues saves a JSON array for every repository, each object exactly as GitHub returns it (the excerpt below shows only a few of its ~40 properties):

• Issue number, title, body, state (open/closed), state_reason, locked
• Labels, assignees, milestone
• Created/updated/closed timestamps
• Author (full user object), author_association, reactions
• URL and HTML URL

GitHub's issues API lists pull requests as issues too; they are recognisable by their pull_request property. Issues and pull requests share one number space.

--issue-comments saves, for every issue and every pull request, an array of its comments (body, author, timestamps). For a pull request this is the conversation thread under the description; inline review comments are --pull-comments.

--issue-events saves, for every issue and pull request, its events from /issues/<n>/events: closed/reopened, labeled/unlabeled, assigned, milestoned, renamed, locked, merged, ... (GitHub's separate /timeline, with cross-references and commits, is not fetched.)

Output structure

json/repos/<repo>/
├── issues.json              ← array of issue objects (pull requests included)
├── issue_comments/<n>.json  ← comments of issue or pull request <n>
└── issue_events/<n>.json    ← events of issue or pull request <n>

JSON schema excerpt

[
  {
    "number": 1,
    "title": "Found a bug",
    "state": "closed",
    "body": "I found a bug...",
    "user": { "login": "octocat" },
    "labels": [{ "name": "bug" }],
    "created_at": "2022-01-01T00:00:00Z",
    "closed_at": "2022-01-02T00:00:00Z"
  }
]

Pull Requests

Enable PR backup with one or more of:

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --repositories \
  --pulls \
  --pull-comments \
  --pull-commits \
  --pull-reviews

What is backed up

--pulls saves the pull request list objects exactly as GitHub returns them:

• Number, title, body, state (open/closed; a merged PR is closed with a non-null merged_at)
• Head and base branch / commit SHA, _links
• Assignees, labels, milestone, requested reviewers, auto_merge, draft
• merge_commit_sha

The list endpoint does not include merge statistics (merged, commits, additions, deletions, changed_files); those exist only on the single-pull-request endpoint, which is not fetched, and they are not written.

--pull-comments saves review comments (inline code comments attached to specific lines, with their diff hunk and reply threading).

--pull-commits saves the list of commits included in each PR (GitHub returns at most 250).

--pull-reviews saves review decisions (approved, changes requested, dismissed) and review bodies.

The conversation thread and the events of a pull request are written by --issue-comments and --issue-events (see above).

Output structure

json/repos/<repo>/
├── pulls.json               ← array of PR objects
├── pull_comments/<n>.json   ← review comments of pull request <n>
├── pull_commits/<n>.json    ← commits of pull request <n>
└── pull_reviews/<n>.json    ← reviews of pull request <n>

API calls per repository

The per-item calls are made on the first run and, on later runs, only for issues and pull requests that changed since the repository's watermark (see Incremental runs; --full repeats all of them). The listing calls cost one request per 100 items on every run. Consider the rate-limit budget (5 000 requests per hour per token) for large organisations; Rate limit explains what happens when it is exhausted.

Releases & Assets

Backup Release Metadata

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --releases

This saves a JSON array for each repository, each release exactly as GitHub returns it:

• Tag name and target_commitish
• Release title and body (markdown)
• Author and timestamps
• List of assets (filename, size, digest, download count, URL, uploader)
• Whether the release is a draft or prerelease

Output

json/repos/<repo>/releases.json

JSON schema excerpt

[
  {
    "id": 12345,
    "tag_name": "v1.0.0",
    "name": "Version 1.0.0",
    "body": "## What's Changed\n...",
    "draft": false,
    "prerelease": false,
    "created_at": "2023-06-01T00:00:00Z",
    "published_at": "2023-06-01T12:00:00Z",
    "author": { "login": "octocat" },
    "assets": [
      {
        "name": "app-linux-x86_64.tar.gz",
        "size": 5242880,
        "download_count": 1234,
        "browser_download_url": "https://github.com/..."
      }
    ]
  }
]

Download Release Assets

Warning: Binary release assets can be very large. Assess disk space before enabling.

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --releases \
  --release-assets

--release-assets requires --releases to be set.

Assets are downloaded and stored alongside the JSON metadata:

json/repos/<repo>/
├── releases.json
└── release_assets/
    └── v1.0.0/
        ├── app-linux-x86_64.tar.gz
        ├── app-linux-x86_64.tar.gz.sha256
        ├── app-darwin-arm64.tar.gz
        ├── app-darwin-arm64.tar.gz.sha256
        ├── checksums.txt
        └── checksums.txt.sha256

How downloads are made safe

• Streamed, not buffered. Each asset is written to disk as it arrives (to .<file>.part in the same directory) and renamed into place when complete, so memory use does not depend on the asset size and a crash, kill or full disk never leaves a truncated file under the real name.
• Verified. The byte count must equal the size GitHub reports, and the SHA-256 must equal GitHub's digest when the release provides one. A download that fails either check is discarded and reported; the remaining assets are still attempted.
• Resumable by re-running. An asset already on disk is skipped only if its size equals GitHub's size and its content matches the API digest (or, if GitHub has none, the checksum in its .sha256 sidecar). A truncated or altered file is downloaded again.
• Token stays with GitHub. GitHub redirects asset downloads to its storage host. The Authorization header is sent to GitHub only; a redirect to another host, port or scheme is followed without the token, and a redirect from HTTPS to plain HTTP is refused.

Combining with S3

For large assets, sync them to S3 after backup:

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --releases --release-assets \
  --s3-bucket my-bucket \
  --s3-include-assets

By default, --s3-bucket only syncs JSON metadata. Add --s3-include-assets to also upload the binary assets.

Storage Estimates

Gists & Wikis

Gists

GitHub Gists are small snippets or files hosted on gist.github.com. Each gist has its own git repository.

Backup Owned Gists

github-backup octocat --token $GITHUB_TOKEN --output /backup --gists

Clones all gists owned by octocat as bare mirror repositories:

git/gists/<gist-id>.git/
json/gists/<gist-id>.json

The JSON file contains the gist object exactly as GitHub returns it:

• Gist ID, description, visibility (public/secret)
• List of files (filename, language, size)
• Owner, created/updated timestamps
• Comment count
• Git URLs (HTTPS and SSH)

json/gists/index.json lists all owned gists.

Secret gists. GitHub's public gist listing for a user never contains secret gists. When the token belongs to the account being backed up (the login is compared case-insensitively), GET /gists is used and secret gists are included. For any other user, without a token, or with a token that cannot call GET /user (GitHub App tokens), only public gists are backed up.

Backup Starred Gists

github-backup octocat --token $GITHUB_TOKEN --output /backup --starred-gists

Records the gists starred by the authenticated user (not necessarily octocat). This requires the gist OAuth scope.

Only the gist metadata is saved (json/gists/<gist-id>.starred.json and json/gists/starred_index.json); starred gists are not cloned, so their file contents are not part of the backup.

Combined

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --gists --starred-gists

Gist update behaviour

On subsequent runs, gists are updated in place with git fetch --all (with --prune added if --prune is set), exactly like mirror clones of repositories. Gists are always mirror clones; --clone-type does not apply to them. A gist that cannot be cloned or updated is recorded as a failure of the run (gist <id>) while the other gists continue.

Wikis

GitHub repository wikis are stored as separate git repositories (the <repo>.wiki.git URL).

Backup Wikis

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --repositories --wikis

--wikis runs independently of --repositories — you do not need to enable --repositories to back up wikis, but both flags are commonly used together.

Output

git/wikis/<repo>.wiki.git/

Each wiki is cloned as a bare mirror. The repository contains all wiki pages as Markdown files, plus the full commit history.

Notes

• Repositories that have no wiki will be skipped silently.
• A wiki must be initialised (have at least one page) before it can be cloned.
• Private repository wikis require a token with the repo scope.

User & Organisation Data

These categories back up list-style metadata that is associated with the owner rather than individual repositories.

Starred Repositories (JSON list)

github-backup octocat --token $GITHUB_TOKEN --output /backup --starred

Saves a JSON array of every repository the owner has starred:

json/starred.json

Each entry is the repository object exactly as GitHub returns it (name, description, owner, visibility, star and fork counts, topics, license, clone URLs and everything else GitHub includes).

Clone Starred Repositories

github-backup octocat --token $GITHUB_TOKEN --output /backup --clone-starred

Clones (or updates) every starred repository as a bare mirror. This is independent from --starred — you can use either or both.

Repositories are cloned into:

git/starred/<upstream-owner>/<repo-name>.git

Durable Queue and Refresh

--clone-starred uses a durable JSON queue stored at:

json/starred_clone_queue.json

The queue is written atomically after every repository, and processed one repository at a time. Two situations:

• A pass is in progress (some item is still pending, for example because the run was interrupted by Ctrl+C, a power cut or a network loss): re-running the same command resumes and leaves the finished items alone.
• The previous pass is complete (nothing is pending): the next run starts a new pass: every item, including earlier failures, goes back to pending with a fresh retry budget, so each run refreshes every starred mirror (git fetch) and retries what failed. A large starred list therefore costs a full pass on every run.

# First run: clones whatever it can, writes progress to the queue
github-backup octocat --output /backup --clone-starred

# Interrupted?  Re-run the same command: it resumes where it left off
github-backup octocat --output /backup --clone-starred

Newly starred repositories are merged into the queue on every run.

Retry & Backoff

Failed clones are retried up to 4 total attempts with exponential backoff:

After 4 failures the item is marked "failed" in the queue and counted as a failure of the run (exit status 3, step: "starred clone <owner>/<repo>" in the report). It is retried automatically in the next pass; there is nothing to edit by hand.

Progress Logging

After each clone the tool emits a structured log line:

INFO starred repo cloned repo="rust-lang/rust" done=42 pending=1505 failed=0 total=1547 rate_per_min="8.3" eta_secs=10880

Clone Type

The clone mode follows the same --clone-type flag as owned repos (default: mirror; --lfs is not applied to starred repositories). Use --prefer-ssh to clone via SSH instead of HTTPS.

Not included in --all

--clone-starred is deliberately not enabled by --all because it can consume significant disk space and run time for users with hundreds or thousands of starred repositories. Add it explicitly (--all --clone-starred works) when needed. It is skipped in a --dry-run.

Watched Repositories

github-backup octocat --token $GITHUB_TOKEN --output /backup --watched

Saves repositories the owner is watching (not just the automatic watch on own repos):

json/watched.json

Followers

github-backup octocat --token $GITHUB_TOKEN --output /backup --followers

Saves the list of accounts that follow the owner:

json/followers.json

Following

github-backup octocat --token $GITHUB_TOKEN --output /backup --following

Saves the list of accounts the owner follows:

json/following.json

Organisation Members

Organisation targets only — this flag has no effect when backing up a regular user account.

github-backup my-org --org --token $GITHUB_TOKEN --output /backup --org-members

Saves the member list of the organisation that the token can see (members whose membership is private are listed only to a token that may see them):

json/org_members.json

Each entry is the user object exactly as GitHub returns it (login, ID, avatar URL, profile URL, ...).

Organisation Teams

Organisation targets only — this flag has no effect when backing up a regular user account.

github-backup my-org --org --token $GITHUB_TOKEN --output /backup --org-teams

Saves all teams within the organisation, including nested parent–child relationships:

json/org_teams.json

Each entry includes the team's name, slug, description, privacy setting, permission level, member and repository URLs, and an optional parent field for nested teams.

All User Data Together

github-backup octocat --token $GITHUB_TOKEN --output /backup \
  --starred --watched --followers --following

For an organisation backup with all owner-level data:

github-backup my-org --org --token $GITHUB_TOKEN --output /backup \
  --starred --watched --followers --following \
  --org-members --org-teams

Or simply use --all to include everything.

Organisation Notes

When using --org, all flags are available but some apply only to organisation targets. --starred, --watched, --followers and --following call GitHub's /users/<owner>/... endpoints with the organisation's name; what GitHub returns for an organisation account has not been verified by the project:

Local Filesystem Storage

By default github-backup writes everything to the local filesystem. The root directory is set with --output (or output in the config file) and defaults to the current directory.

Output Directory Layout

Each owner gets its own directory below --output:

<output>/
├── .github-backup.lock
└── <owner>/
    ├── git/                ← clones: repos/, wikis/, gists/, starred/
    └── json/               ← JSON metadata and the bookkeeping files

The complete tree, with the file each flag produces, is on the Output Directory Layout page. It is the only place that lists every file; this page does not repeat it.

Incremental Updates

github-backup is designed to be run repeatedly into the same directory. On later runs:

• Git repositories are updated in place (git fetch --all for mirrors) instead of being cloned again. Deleted branches and tags are kept unless --prune is given; force-pushed branches are overwritten.
• JSON lists (issues.json, pulls.json, releases.json, ...) are fetched in full and written again. issues.json and pulls.json are merged with what is already stored, so an item that has since disappeared from GitHub's listing stays in the backup.
• Per-item files (issue_comments/<n>.json, ...) are fetched again only for items that changed since the repository's watermark; see Incremental runs.
• Release assets are kept when they are complete (size and checksum match) and downloaded again otherwise.

Every file is written to a temporary name and renamed into place, and a fresh clone is made in a hidden staging directory and renamed when it is complete, so an interrupted run does not leave a truncated file or a half-written clone under its real name.

Disk Space

Plan for the repositories plus their metadata. A rough guide:

A mirror also keeps refs/pull/*, which makes it larger than a plain clone. The tool does not check free space before a run (--doctor does not either); measure with du -sh <output>/<owner> after a trial run. A full disk stops the run with exit status 1.

Permissions

The process needs read and write access to --output and must be able to run git (and git-lfs with --lfs). New files get the mode implied by the process umask. Backups can contain private repository code, webhook configuration and security advisories, so restrict the directory (chmod 700) or encrypt the volume.

For unattended runs create a dedicated user:

sudo useradd -r -m -d /var/backup/github github-backup
sudo -u github-backup env GITHUB_TOKEN="$GITHUB_TOKEN" \
  github-backup octocat --output /var/backup/github --all

When the tool runs git in a repository it trusts exactly that path (-c safe.directory=<path>), so a backup directory that was created by another user, or chown-ed by a file manager or NAS tool, still updates. Your own git commands in such a directory may stop with "detected dubious ownership"; fix the ownership (chown -R) or add the path with git config --global --add safe.directory <path>.

Getting the Code Back

A mirror clone is a valid git repository:

# Clone from the backup
git clone /var/backup/github/octocat/git/repos/Hello-World.git hello-world

To push it to a new remote, see Restore: do not use git push --mirror against GitHub, which rejects the refs/pull/* refs a mirror contains.

S3-Compatible Storage

github-backup can sync the JSON metadata of a backup (issues, pull requests, releases, labels, ...) and, optionally, release assets to any S3-compatible object store after the local backup completes.

Repository clones are not uploaded. The bare git mirrors, wikis and gist clones under <output>/<owner>/git/ stay local. For an offsite copy of the code itself use --mirror-to (see Mirroring) or a tool that copies directories (rclone sync, restic, borg).

If any upload or deletion fails, the run is recorded as incomplete: it exits with status 3 (see exit codes), the summary, --report, metrics and webhook ("partial") all list the failure, and the log says which objects failed and why. A failed S3 sync never undoes the local backup, which is already complete at that point.

Supported Providers

The endpoint must include its scheme (https://...). A bare host name is assumed to be https:// and logged as a warning.

Custom endpoints use path-style addressing (https://host/bucket/key). On AWS the bucket is addressed in the host name; a bucket name containing dots falls back to path-style on the regional endpoint.

Verification status. The S3 client is tested against an in-process fake server that verifies AWS Signature Version 4 independently and reproduces the four signature examples published by AWS. No real AWS, MinIO, B2, R2, Wasabi or Spaces server was available when this was written, so provider-specific behaviour is not verified by the project's tests. If a provider rejects requests, the error message carries the server's code and message; please report it.

Basic Usage

The bucket must already exist; github-backup never creates buckets.

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

github-backup octocat --output /backup --all \
  --s3-bucket my-backup-bucket \
  --s3-region us-east-1

Prefer the environment variables over --s3-access-key / --s3-secret-key: command-line values are visible to every user of the machine in the process list. Temporary credentials work too: set AWS_SESSION_TOKEN (or --s3-session-token) next to the key pair; it is signed as x-amz-security-token. AWS_PROFILE, ~/.aws/credentials, instance metadata and web-identity credentials are not read.

If --s3-bucket is set but a key is missing, the run reports exactly which credential is missing.

Provider-Specific Examples

Backblaze B2

export AWS_ACCESS_KEY_ID="$B2_KEY_ID" AWS_SECRET_ACCESS_KEY="$B2_APP_KEY"
github-backup octocat --output /backup --all \
  --s3-bucket my-b2-bucket \
  --s3-region us-west-004 \
  --s3-endpoint "https://s3.us-west-004.backblazeb2.com"

Cloudflare R2

export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$R2_SECRET_KEY"
github-backup octocat --output /backup --all \
  --s3-bucket my-r2-bucket \
  --s3-region auto \
  --s3-endpoint "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"

MinIO (self-hosted)

Create the bucket first (mc mb local/github-backup).

export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin
github-backup octocat --output /backup --all \
  --s3-bucket github-backup \
  --s3-region us-east-1 \
  --s3-endpoint "http://minio.internal:9000"

A plain http:// endpoint sends object names and data unencrypted; a warning is logged unless it is a loopback address.

Private certificate authority

The TLS client trusts the system certificate store. For a self-signed MinIO set SSL_CERT_FILE=/path/to/ca.pem (or SSL_CERT_DIR) for the process.

Proxies

HTTPS_PROXY and NO_PROXY apply to GitHub API calls only. S3 traffic connects to the endpoint directly; behind a proxy that blocks it, the upload phase fails with "cannot reach the endpoint".

Object Layout and Key Prefix

<prefix>/<owner>/json/<path relative to the json directory>

For example, --s3-prefix backups/github/ and owner octocat give backups/github/octocat/json/repos/hello/issues.json. Because the owner is part of every key, several owners can share one bucket and prefix without overwriting each other. The prefix may be empty and a trailing slash is optional.

Including Release Assets

By default only JSON metadata is synced. Add --s3-include-assets to also upload release assets (.../release_assets/<tag>/<file>):

--s3-bucket my-bucket --s3-include-assets

Warning: Binary assets can be large; each file is read into memory and uploaded (in 16 MiB parts above that size). Use a lifecycle rule to manage costs. Dropping --s3-include-assets later does not delete assets that are already in the bucket, not even with --s3-delete-stale.

Incremental Sync

Every uploaded object carries the SHA-256 of the local file as user metadata (x-amz-meta-sha256). On the next run the client asks the server for the object (HEAD) and skips the upload only when that digest and the size match. A file edited without changing its size is uploaded again. An object that has no digest (uploaded by another tool or an older version) is uploaded again once.

For encrypted uploads the stored value is a keyed digest (HMAC-SHA256 under a key derived from the encryption key), so the bucket never learns an unkeyed hash of your plaintext, and a new key makes every object upload again. See the Encryption guide.

If the credentials may not HEAD objects (no s3:GetObject, or no s3:ListBucket so a missing key reads as 403), the client cannot detect unchanged files and uploads everything on every run; a warning says so.

Deleting Stale Objects (--s3-delete-stale)

--s3-delete-stale removes objects under <prefix>/<owner>/json/ whose local file no longer exists. Safety rules:

• The listing covers <prefix>/<owner>/json/ only (with the trailing slash), so neighbouring prefixes such as github-backup-old/ are never touched.
• Nothing is deleted when the backup run had failures, when any part of the local tree could not be read, when the local tree holds no files, when any upload failed in the same run, or when the sync was aborted. The reason is logged.
• Release assets that were only left out of this run (no --s3-include-assets) are never considered stale.
• --dry-run never reaches this stage: a dry run skips the S3 sync altogether (see Dry Run).
• After you switch encryption on, the old plaintext objects are removed by --s3-delete-stale (they would otherwise stay readable in the bucket).

Deletion is permanent. On a bucket where you use this flag, enable bucket versioning (with a lifecycle rule that expires old versions): the tool deletes with a plain DeleteObject, so a wrongful delete leaves a recoverable version behind a delete marker.

Dry Run

--dry-run skips the S3 sync completely: the bucket is not contacted and nothing is uploaded or deleted. The sync code has a read-only preview mode (HEAD and LIST requests only), but the command line does not use it, so there is currently no way to preview an upload. To check credentials and bucket access before a real run, upload a small test with a throw-away prefix (--s3-prefix test/) and delete it afterwards.

Required Permissions

Minimum IAM policy (replace the bucket and prefix):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-backup-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["github-backup/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::my-backup-bucket/github-backup/*"
    }
  ]
}

• s3:ListBucket (on the bucket) is needed for the stale-object listing and, importantly, so that HEAD of a not-yet-uploaded key answers 404 instead of 403.
• s3:GetObject is what authorises HEAD; there is no s3:HeadObject action.
• s3:DeleteObject is only used with --s3-delete-stale; the multipart actions only for objects above 16 MiB.

Errors

A failed run reports the HTTP status, the S3 error code and message, and a hint:

Transient problems (5xx, SlowDown, dropped connections, stalls) are tried up to 5 times in total (the first attempt and four retries) with exponential back-off and jitter. A request is only abandoned when no data moved for 60 seconds, so a slow upload that keeps progressing is not cut off. A configuration-class error (bad credentials, missing bucket) stops the run after the first failure instead of repeating it for every file. Incomplete multipart uploads are aborted on any failure.

At-Rest Encryption

export BACKUP_ENCRYPT_KEY=$(openssl rand -hex 32)
github-backup octocat --output /backup --all --s3-bucket my-bucket

Encrypted objects receive a .enc suffix. See the Encryption guide for the wire format, what is and is not protected, key rotation and decryption.

S3 Flags Reference

The config file can set the bucket, region, prefix, endpoint, keys and s3_include_assets; --s3-delete-stale, the session token and the encryption key come from the command line or the environment.

Not Covered

--doctor / --check do not test S3 access, and --dry-run does not either. The TUI has no S3 settings: S3 sync is available from the command line and the config file only. Repository clones, wikis and gists are never uploaded; to copy them off-site use --mirror-to, or copy <output>/<owner>/git/ with a tool such as rclone sync, restic or borg.

Implementation Details

The S3 client uses a pure-Rust SigV4 implementation (no AWS SDK, no openssl). Signatures are HMAC-SHA256 computed with the sha2 and hmac crates; every request signs exactly the headers it sends. At-rest encryption uses aes-gcm from the same RustCrypto family.

At-Rest Encryption (AES-256-GCM)

github-backup can encrypt every file before it is uploaded to S3 with AES-256-GCM, using the RustCrypto aes-gcm crate. Encryption is optional and applies only to S3 uploads; local files are never encrypted by the tool (use LUKS or similar for that). The crate is not a FIPS-validated module; the algorithm and mode are standard (NIST SP 800-38D).

Quick Start

export BACKUP_ENCRYPT_KEY=$(openssl rand -hex 32)   # 64 hex characters
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...

github-backup octocat --output /var/backup/github --all \
  --s3-bucket my-github-backups

Store the key somewhere other than the machine that holds the backups (a password manager, a secrets manager). If you lose the key, the encrypted objects cannot be recovered.

What Is and Is Not Protected

• Protected: the contents of every uploaded file, including tamper detection (any change to an object makes decryption fail).
• Not protected: object names (they contain your private repository names, for example .../repos/<repo>/issues.json.enc), object sizes (plaintext size + 28 bytes), and upload timing. Anyone who can list the bucket sees those.
• Not protected: swapping and rollback. No associated data is authenticated, so someone with write access to the bucket (but not the key) can swap two encrypted objects, or put back an older valid version of one; decryption accepts both. They cannot forge new content. Use bucket versioning / object lock if that matters to you.
• Local files and git clones are not encrypted by this feature (and clones are not uploaded at all).
• Enabling encryption later does not remove the earlier plaintext objects. They stay readable in the bucket until you delete them; running once with --s3-delete-stale removes the plaintext objects that now have an encrypted counterpart, or delete them yourself.

Key Management

The key is not written to disk by the tool and error messages never contain any of its characters. What is wiped from memory: the decoded 32-byte key buffer, the copy shared by the upload tasks, and the AES round keys inside the cipher (the aes crate's zeroize feature is enabled). What is not wiped: the hex string held by the argument parser, the process environment and command line, and any copy the operating system makes (swap, core dumps).

The key is used directly as the AES-256 key (no key derivation); if you derive it from a passphrase, use a KDF such as Argon2id first. A subkey derived from it with HKDF-Expand (RFC 5869) is used for the upload digest below, so the AES key itself is never used with another primitive.

Wire Format

┌──────────────────────┬────────────────────────────────┐
│  12-byte random nonce│  ciphertext + 16-byte GCM tag  │
└──────────────────────┴────────────────────────────────┘

• A fresh random 96-bit nonce (operating-system CSPRNG) is generated for every object; encrypting the same content twice gives different ciphertext.
• The tag authenticates the ciphertext; there is no additional authenticated data.
• Encrypted objects get a .enc suffix (labels.json becomes labels.json.enc).
• The whole file is encrypted as a single message, so it is held in memory while it is processed. NIST allows one key to encrypt up to 2^32 random-nonce messages (collision probability about n^2 / 2^97); a backup tool stays many orders of magnitude below that. A single message is limited to about 64 GiB by the cipher implementation.

Skipping Unchanged Files, and Key Rotation

Each encrypted object carries x-amz-meta-sha256, which for encrypted uploads is HMAC-SHA256 of the plaintext under a key derived from your encryption key, never a plain hash. Consequences:

• An unchanged file is skipped; a changed file (even with the same size) is uploaded again.
• Someone with read access to the bucket learns that two plaintexts are equal (equal digests) but cannot test a guess about their content.
• Changing the key changes every digest, so the next run uploads every file again, encrypted under the new key, overwriting the old objects.

Rotating the key

There is no automatic re-encryption of what is already in the bucket, but the steps are short:

1. Generate a new key and keep the old one until the end.
2. Run the backup with the new key (BACKUP_ENCRYPT_KEY=<new>). Every file whose local copy still exists is uploaded again under the new key.
3. Verify: decrypt one or two objects with the new key (see below).
4. Objects whose local file no longer exists (and, with --s3-delete-stale not set, anything removed locally) remain encrypted under the old key. Either run once with --s3-delete-stale to remove them, or keep the old key for as long as you need them.
5. Retire the old key only when no object remains that needs it, and only after step 3 succeeded. Bucket versions (if versioning is on) of the old objects stay encrypted under the old key.

Decrypting

Use the tool itself (works on the objects this version and earlier versions produced):

export BACKUP_ENCRYPT_KEY=...   # the key used for the upload
github-backup --decrypt \
  --decrypt-input issues.json.enc --decrypt-output issues.json

--decrypt needs neither an OWNER nor network access, only the key and the two file paths. Download objects with your provider's tool first (the tool itself has no download command), then decrypt one file at a time; for many files loop in the shell.

Without the tool, any AES-GCM implementation can do it. openssl enc cannot: it refuses AEAD ciphers ("AEAD ciphers not supported"). With Python and the cryptography package:

from cryptography.hazmat.primitives.ciphers.aead import AESGCM

key = bytes.fromhex("your_64_hex_char_key")
with open("labels.json.enc", "rb") as f:
    blob = f.read()
plaintext = AESGCM(key).decrypt(blob[:12], blob[12:], None)
open("labels.json", "wb").write(plaintext)

This snippet was run against a file produced by the tool's encrypt() (key 42 repeated 32 times, plaintext {"hello":"world"}\n, 46-byte object):

46 b'{"hello":"world"}\n'

Security Notes

• AES-256-GCM gives 256-bit key strength and authenticated encryption; any change to the ciphertext or tag makes decryption fail.
• Random nonces come from OsRng.
• You can combine this with server-side encryption (SSE-S3 / SSE-KMS) for defence in depth; that protects the disks, not the names.

Push Mirrors (Gitea, Codeberg, Forgejo, GitLab)

After the local backup, github-backup can push every cloned repository to a second git host, so that a copy of the code exists somewhere that is not your backup disk. (S3 never receives git data; this is the off-site path for code.)

Supported Destinations

GitHub itself is not a supported destination for --mirror-to. (Pushing a backup to a new GitHub repository is a manual step; see the Restore Guide.)

Basic Usage

export GITHUB_TOKEN=ghp_dummy_token
export MIRROR_TOKEN=dummy_codeberg_token

github-backup octocat \
  --output /backup \
  --repositories \
  --mirror-to https://codeberg.org \
  --mirror-owner your_codeberg_username

For each repository found under <output>/octocat/git/repos/ the tool:

1. checks whether the repository exists at the destination and creates it if not;
2. verifies that it may write to it (see Safety);
3. pushes all branches and tags, and deletes destination branches and tags that no longer exist locally (git push --prune with the refspecs +refs/heads/*:refs/heads/* and +refs/tags/*:refs/tags/*).

This happens after the local backup of the same run. Under --dry-run it is skipped.

What Is and Is Not Pushed

• Branches and tags: yes, force-updated, with deletions mirrored.
• refs/pull/* and other namespaces a GitHub mirror clone contains: no. GitHub-style hosts reject them, which makes a plain git push --mirror fail as a whole.
• Only directories ending in .git under git/repos/: mirror, bare and shallow clones and --lfs. Working-tree clones (--clone-type full, stored as <repo>/) are not pushed. Wikis, gists and starred clones are not pushed either.
• LFS objects are not pushed.
• Issues and other metadata are not pushed; only git data.

Safety

• A repository is created private by default. --mirror-public makes the mirror of a repository public, but only if GitHub's listing (repos.json) says the source is public: a private source, or one whose visibility is unknown (no repos.json), is created private whatever the flags say. --mirror-private states the default explicitly. An existing repository keeps its current visibility.
• A repository this tool did not create is never pushed into. Every repository it creates gets the description GitHub mirror of <owner>/<repo>. An existing destination repository is accepted only if it carries that description or is still empty; otherwise the push is refused for that repository and recorded as a failure. (If you want to mirror into an existing, non-empty repository, clear it first or set that description yourself.)
• The token never appears in a command line or a file. It is handed to the git child in an environment variable and answered by a credential helper that is limited to the destination's scheme, host and port.

Mirror Flags

--mirror-to, --mirror-token (or MIRROR_TOKEN), --mirror-owner, --mirror-private and --mirror-public can also be set in the config file; --mirror-type cannot.

Gitea, Forgejo, Codeberg

--mirror-owner may be your own user name or an organisation: when it differs from the login of the token, repositories are created through the organisation endpoint (POST /api/v1/orgs/<owner>/repos), otherwise through POST /api/v1/user/repos. The token needs permission to create repositories and to push (on Codeberg, create it under Settings → Applications with the write:repository permission, and organisation permissions when --mirror-owner is an organisation).

GitLab

github-backup octocat --output /backup --repositories \
  --mirror-to https://gitlab.com \
  --mirror-type gitlab \
  --mirror-owner my-group-or-username

--mirror-owner is the namespace path (a user or a group). The token must be allowed to create projects there (a personal or group access token with the api scope).

Failures

A destination that cannot be reached, a refused push or a foreign repository is logged per repository and recorded as a failure of the run (step mirror push, exit status 3); the remaining repositories are still mirrored. The destination's own API calls do not use HTTPS_PROXY (the git push itself follows git's proxy settings).

Docker Compose Example

services:
  backup:
    image: ghcr.io/tomtom215/github-backup-rust:latest
    environment:
      GITHUB_TOKEN:
      MIRROR_TOKEN:
    command: >
      octocat
      --output /backup
      --repositories
      --mirror-to https://codeberg.org
      --mirror-owner your_username
    volumes:
      - backup_data:/backup

volumes:
  backup_data:

Export GITHUB_TOKEN and MIRROR_TOKEN in the shell (or an .env file) that runs Compose. See the Docker guide for the bundled Compose profiles (codeberg, gitlab).

Limitations

• One-way: GitHub to destination. Anything pushed to a mirror by hand on a branch that exists on GitHub is overwritten at the next run.
• The first push of a large repository can take a long time. The push has no stall timeout of its own.
• The behaviour against real Codeberg, Gitea, Forgejo and GitLab servers has not been verified by the project; it is tested with mock servers and local git remotes.

CLI Reference

Reference for every github-backup command-line flag. github-backup --help is the authoritative list for the binary you have installed; this page is checked against it, and the Env var and Default columns are taken from it.

Synopsis

github-backup [OPTIONS] [OWNER]
github-backup [OWNER] --tui
github-backup --config <FILE> [OPTIONS]
github-backup --completions <SHELL>
github-backup --print-config-template

--completions <SHELL> is not listed by --help (it is handled before the normal argument parsing); --help and --version (-h, -V) behave as usual.

Arguments

Authentication

With no credential at all the tool runs unauthenticated (public data, 60 requests per hour). It refuses to start, with exit status 1, when one of --private, --hooks, --deploy-keys, --collaborators, --org-members, --org-teams, --actions, --action-runs, --packages, --discussions or --projects is given explicitly. That check does not look inside --all: an anonymous --all starts, and everything that needs a token fails or comes back empty.

GitHub Enterprise Server

See GitHub Enterprise Server.

Configuration

Output

Target Type

Broad Selector

Repository Options

Issue and Pull Request Options

Repository Metadata

GitHub Actions and Environments

Discussions, Classic Projects, Packages

Organisation Data

User and Organisation Data

Repository Filters

Patterns match the repository name only (not owner/name): * matches any sequence, ? one character, case-insensitively. There is no pattern for "archived" repositories; *archived* matches names that contain the word.

github-backup octocat --output /backup --repositories --include-repos "rust-*"
github-backup octocat --output /backup --repositories --exclude-repos "*-old,*-fork"

Incremental Behaviour

Issue and pull request lists are always fetched in full and merged into issues.json / pulls.json; a run never shrinks them. What is incremental is the per-item data (comments, events, commits, reviews): a repository's own watermark from the previous clean run decides which items can be skipped. See Incremental runs.

Push Mirror Options

See Mirroring.

S3 Options

At-Rest Encryption

Restore

See the Restore guide.

Manifest and Verify

Monitoring

Execution Options

What --dry-run does

A dry run fetches the repository list (read-only API calls) and logs which repositories it would back up. It writes nothing: no files or directories (not even the lock), no state, report, manifest, metrics, history, no webhook, no S3 upload and no mirror push, and it runs no git. Owner-level data and gists are skipped. With --restore it makes no API call at all.

Deprecated

The tool keeps one continuously updated backup per owner under <output>/<owner>/; it never deletes snapshot directories. Rotate or expire copies with a tool built for it (restic, borg, ZFS or LVM snapshots, a lifecycle rule on the bucket). github-backup --help still describes the two flags as if they pruned directories; they do not.

Logging

RUST_LOG overrides the level; see Environment Variables.

Diagnostics

--doctor runs, in order: the git binary and its version, whether the output directory exists or can be created (it is created if missing) and is writable, the kind of credential (classic, fine-grained, OAuth or app token, judged by its prefix), whether the API answers, and whether GitHub accepts the token (GET /rate_limit, which costs no quota). A rejected token (HTTP 401 or 403) or an unreachable API is a failure; a server without /rate_limit is only a warning. It does not check free disk space, the token's scopes, S3 access, the mirror destination or git-lfs. It exits 0 when no check failed and 1 otherwise.

--list-scopes maps each enabled flag to the scopes it would need. With --all it prints only public_repo repo (it does not expand --all), and it recommends user:follow for --followers and admin:public_key for --deploy-keys, which only read data. For a token covering --all use the table in Authentication.

Interactive TUI

--tui starts the full-screen interface instead of a run. The owner, token, output directory and API URL given on the command line pre-fill the form; other flags are not read. See the Interactive TUI guide.

Shell Completions

--completions <SHELL> prints a completion script and exits without a token or network access. Supported shells: bash, zsh, fish, powershell, elvish. Per-shell setup is in Installation.

Exit Codes

Failures never stop the rest of the run: a repository that cannot be cloned does not prevent its issues or the other repositories from being backed up. Each failure is recorded once and shows up in the log, the summary banner, the --report file (failure_count, failures[]), the metrics (github_backup_failures, github_backup_success), backup_history.json and the webhook ("status": "partial"); they cannot disagree with the exit code. A failure of the notification itself (webhook, report or metrics file that cannot be written) is logged but does not change the exit code.

Config File (TOML)

github-backup supports a TOML configuration file, making it easy to manage complex backup configurations without long command lines.

Loading a Config File

github-backup --config /etc/github-backup/config.toml

Or with the short form:

github-backup -c /etc/github-backup/config.toml

Precedence

The file supplies defaults; the command line and the environment are applied on top. The exact rules, because "the command line always wins" is not true for every kind of setting:

# Config: owner = "octocat", concurrency = 4
# Override concurrency for this run only:
github-backup --config config.toml --concurrency 16

A config file that cannot be read or contains an unknown key is an error (exit status 1): every key is checked, so a typo such as issuez = true is reported instead of silently ignored.

Full Config File Example

# /etc/github-backup/config.toml

# ── Identity ───────────────────────────────────────────────────────────────
owner = "octocat"

# Authentication (prefer GITHUB_TOKEN environment variable instead)
# token = "ghp_xxx"

# Output directory
output = "/var/backup/github"

# Parallelism
concurrency = 8

# Target type (default: user)
# org = true

# ── Clone behaviour ────────────────────────────────────────────────────────
# clone_type = "mirror"  # "mirror", "bare", "full", "shallow:<depth>" or { shallow = <depth> }
# prefer_ssh  = false
# lfs         = false
# prune       = false   # true: delete refs that were deleted on GitHub

# ── Backup categories ──────────────────────────────────────────────────────

# Enable the categories of `--all` (clone_starred and action_runs stay opt-in)
# all = true

# Or enable individually:
repositories     = true
forks            = false
private          = true

issues           = true
issue_comments   = true
issue_events     = false

pulls            = true
pull_comments    = true
pull_commits     = false
pull_reviews     = true

labels           = true
milestones       = true
releases         = true
release_assets   = false

hooks            = false
security_advisories = true
wikis            = true

starred          = true
watched          = false
followers        = false
following        = false
gists            = true
starred_gists    = false

# ── GitHub Actions ─────────────────────────────────────────────────────────
actions          = true
# action_runs   = false  # opt-in; can be large for active repos

# ── Deployment environments ─────────────────────────────────────────────────
environments     = true

# ── Packages ───────────────────────────────────────────────────────────────
# (`discussions` and `projects` are accepted but back up nothing: GitHub's
#  REST API has no endpoint for them.)
packages         = false  # requires the read:packages OAuth scope

# ── Organisation-specific ───────────────────────────────────────────────────
org_members      = false
org_teams        = false

# ── Reporting ─────────────────────────────────────────────────────────────
# report = "/var/log/github-backup/report.json"

# ── Mirror to Gitea/Codeberg ───────────────────────────────────────────────
# mirror_to      = "https://codeberg.org"   # Gitea-type only: mirror_type cannot be set in the file
# mirror_token   = "cb_token"         # or use MIRROR_TOKEN env var
# mirror_owner   = "alice"
# mirror_public  = false   # true: mirrors of public repositories are public
# mirror_private = false   # explicit form of the default

# ── S3-compatible storage ──────────────────────────────────────────────────
# s3_bucket       = "my-github-backup"
# s3_region       = "us-east-1"
# s3_prefix       = "github/"
# s3_endpoint     = ""  # Leave blank for AWS; set for B2/MinIO/R2/etc.
# s3_access_key   = ""  # or use AWS_ACCESS_KEY_ID env var
# s3_secret_key   = ""  # or use AWS_SECRET_ACCESS_KEY env var
# s3_include_assets = false

Minimal Config File

owner = "octocat"
output = "/var/backup/github"
repositories = true
issues = true

Then run:

GITHUB_TOKEN=ghp_xxx github-backup --config config.toml

Full Automated Backup

A complete config for a nightly scheduled backup:

owner       = "my-org"
org         = true
output      = "/var/backup/github"
concurrency = 8
all         = true

# JSON report for monitoring
report = "/var/log/github-backup/report.json"

# Mirror to Codeberg after each run
mirror_to    = "https://codeberg.org"
mirror_owner = "my-org-mirror"

# Sync JSON metadata to S3
s3_bucket = "my-github-backup"
s3_region = "eu-west-1"
s3_prefix = "nightly/"

Run with just:

GITHUB_TOKEN=ghp_xxx MIRROR_TOKEN=cb_xxx \
  AWS_ACCESS_KEY_ID=AKID AWS_SECRET_ACCESS_KEY=SECRET \
  github-backup --config /etc/github-backup/config.toml

Security

• Set file permissions to 0600 to prevent other users reading your token:

  chmod 600 /etc/github-backup/config.toml
  

• Prefer environment variables (GITHUB_TOKEN, MIRROR_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) over storing secrets in the config file, especially in multi-user environments.

Config File Schema

Core

Clone Behaviour

Backup Categories

Mirror Destination

S3-Compatible Storage

GitHub Enterprise Server Config

owner    = "my-org"
org      = true
output   = "/var/backup/github-enterprise"
api_url  = "https://github.example.com/api/v3"
# Needed only when API host and clone host differ (separate load balancers)
# clone_host = "github-git.example.com"
all      = true

Incremental Backups

Nothing needs to be configured: every run records a watermark per repository in <output>/<owner>/json/backup_state.json and the next run uses it (see Incremental runs). The since key is only an override and should normally stay unset; a fixed since date in a file that is used for every run defeats the watermarks.

Repository Filter Config

owner        = "octocat"
output       = "/var/backup/github"
repositories = true

# Only back up repos whose names start with "rust-" or equal "my-tool"
include_repos = ["rust-*", "my-tool"]

# But skip any repo whose name ends in "-archive" (patterns match the repository
# name only; there is no pattern for GitHub's "archived" flag)
exclude_repos = ["*-archive"]

Options That Cannot Be Set in the File

These flags have no config key and must come from the command line (or, where one exists, the environment): --config, --print-config-template, --doctor, --check, --list-scopes, --device-auth, --oauth-client-id, --oauth-scopes, --full, --mirror-type, --s3-session-token, --s3-delete-stale, --dry-run, --manifest, --verify, --keep-last, --max-age-days (deprecated), --prometheus-metrics, --diff-with, --notify-webhook, --history-size, --restore, --restore-target-org, --restore-yes, --encrypt-key, --decrypt, --decrypt-input, --decrypt-output, --quiet, --verbose and --tui. A file that tries to set one of them is rejected as an unknown key.

The Template

github-backup --print-config-template prints an annotated file that lists all 63 keys, commented out. Some of its comments are outdated: it marks output as required (it defaults to .), says that command-line flags always override the file (see Precedence), suggests discussions = true and projects = true (they have no effect) and says that an unset since inherits the previous run's timestamp (each repository has its own watermark).

Environment Variables

github-backup reads several environment variables so that secrets stay out of the command line, shell history and the process list. A variable that is set but empty is treated as unset (container launchers often pass every optional variable as an empty string); surrounding whitespace, such as a newline pasted with a token, is trimmed.

Authentication

GitHub Enterprise

S3 Storage and Encryption

The three AWS_* variables are ignored unless --s3-bucket is given, so credentials exported for other tools never affect a run that does not use S3. AWS_PROFILE, ~/.aws/credentials and instance metadata are not read.

Mirror Push

Notification and Restore

Proxy

The lower-case spelling wins when both are set. The proxy must be an HTTP proxy (http://[user:pass@]host[:port], port 3128 if omitted; credentials are sent as Proxy-Authorization: Basic). https:// targets are reached through an HTTP CONNECT tunnel. A value that is not a usable HTTP proxy URL (socks5://... for example) is ignored with a warning. SOCKS proxies are not supported.

What the variables apply to:

export HTTPS_PROXY=http://proxy.corp.example.com:3128
export NO_PROXY=localhost,.corp.example.com
github-backup octocat --output /backup --all

TLS

Logging

Logs go to stderr; the banners, --doctor, --list-scopes and --print-config-template print to stdout.

# Warnings and errors only
RUST_LOG=warn github-backup ...

# Debug output of the API client only
RUST_LOG=github_backup_client=debug github-backup ...

Container Variables

The image's entrypoint additionally reads GITHUB_OWNER, BACKUP_MODE, BACKUP_FLAGS and UMASK when it is started without arguments; see Docker. They are not read by the github-backup binary.

Setting Variables Securely

For interactive use, read keeps the token out of the shell history:

read -rs GITHUB_TOKEN && export GITHUB_TOKEN

For systemd services use EnvironmentFile= with a 0600 file; for Docker use --env-file or -e GITHUB_TOKEN (no value, so the secret is not in the docker command line); for Kubernetes use a Secret.

[Service]
EnvironmentFile=/etc/github-backup/secrets.env

# /etc/github-backup/secrets.env (mode 0600)
GITHUB_TOKEN=ghp_dummy_token_for_illustration

Output Directory Layout

This page explains the directory and file structure that github-backup creates under --output.

The JSON files are GitHub's own responses

Every *.json file listed below holds the objects exactly as the GitHub API returned them: every property, in GitHub's key order, including the ones this tool does not use itself (reactions, node_id, _links, URL templates, review-comment anchors, ...). Nothing is dropped and nothing is invented (a missing property stays missing; it is not turned into null).

• A list that contains an object the tool cannot interpret (for example a null where an object is expected) is still written in full: that object is appended verbatim after the others and a warning naming the endpoint, position and id/number is logged. One odd object never stops the list or the other categories.
• Files are written deterministically: the same data produces the same bytes, so diffs and checksums are meaningful.
• Expect the JSON to be several times larger than a summary would be.

Files that are lists hold a JSON array. workflows.json, workflow_runs_<id>.json and environments.json hold the array of items (every page of GitHub's {"total_count": ..., "<items>": [...]} response merged); topics.json holds the list of topic names; branch_protections.json is an object keyed by branch name.

Tree

<output>/
├── .github-backup.lock                     ← process lock (stays on disk, see below)
└── <owner>/                                ← GitHub username or org name
    ├── git/                                ← Git repositories
    │   ├── repos/
    │   │   ├── <repo>.git/                 ← mirror clone (default); bare, shallow, --lfs too
    │   │   ├── <repo>/                     ← working tree, with --clone-type full
    │   │   └── …
    │   ├── wikis/
    │   │   ├── <repo>.wiki.git/            ← wiki mirror clone
    │   │   └── …
    │   ├── gists/
    │   │   ├── <gist-id>.git/              ← gist mirror clone
    │   │   └── …
    │   └── starred/                        ← --clone-starred
    │       └── <upstream-owner>/<repo>.git/
    └── json/                               ← JSON metadata
        ├── repos.json                      ← the repositories this backup covers
        ├── starred.json                    ← --starred
        ├── watched.json                    ← --watched
        ├── followers.json                  ← --followers
        ├── following.json                  ← --following
        ├── org_members.json                ← --org-members   (org targets)
        ├── org_teams.json                  ← --org-teams     (org targets)
        ├── packages_<type>.json            ← --packages
        ├── package_versions_<type>_<name>.json
        ├── starred_clone_queue.json        ← --clone-starred progress
        ├── backup_state.json               ← incremental watermarks per repository
        ├── backup_history.json             ← recent runs (--history-size)
        ├── backup_checkpoint.json          ← only after an interrupted run
        ├── backup_manifest.json            ← --manifest
        ├── .backup.lock                    ← engine lock (stays on disk, see below)
        ├── gists/
        │   ├── <gist-id>.json              ← --gists
        │   ├── <gist-id>.starred.json      ← --starred-gists
        │   ├── index.json                  ← all owned gists
        │   └── starred_index.json          ← all starred gists
        └── repos/
            └── <repo>/
                ├── info.json               ← the repository object (always written)
                ├── issues.json
                ├── issue_comments/<n>.json ← one file per issue or pull request
                ├── issue_events/<n>.json   ← one file per issue or pull request
                ├── pulls.json
                ├── pull_comments/<n>.json  ← one file per pull request
                ├── pull_commits/<n>.json
                ├── pull_reviews/<n>.json
                ├── releases.json
                ├── release_assets/
                │   └── <tag>/
                │       ├── <asset-file>        ← binary release asset
                │       └── <asset-file>.sha256 ← its checksum
                ├── labels.json
                ├── milestones.json
                ├── hooks.json
                ├── security_advisories.json
                ├── topics.json
                ├── branches.json
                ├── branch_protections.json ← protected branches (needs admin access)
                ├── deploy_keys.json
                ├── collaborators.json
                ├── workflows.json
                ├── workflow_runs_<id>.json ← one per workflow (--action-runs)
                └── environments.json

discussions.json, discussion_comments_<n>.json, projects.json and project_columns_<id>.json are not produced: GitHub's REST API has no such endpoints, see Discussions and Classic Projects. Repositories that the options exclude (forks without --forks, private repositories without --private, --include-repos / --exclude-repos) get neither a clone nor a metadata directory.

Bookkeeping files

Lock files

Two lock files serialise runs: <output>/.github-backup.lock and <owner>/json/.backup.lock. They are operating-system file locks held for the duration of the run and released by the kernel when the process ends, however it ends (including kill -9, an out-of-memory kill or a power cut). The files themselves are left on disk and are harmless; there is nothing to clean up and no stale-lock state. A second run on the same owner stops with "another backup ... is already running" (exit status 1).

Issue and pull request numbers share one space

GitHub numbers issues and pull requests from the same counter, so #42 is either an issue or a pull request, never both. The per-item files are named by that number:

A pull request appears in issues.json as well as in pulls.json (GitHub's issues API lists pull requests too; they carry a pull_request property).

File Descriptions

Git repositories (git/)

--clone-type bare and shallow:<n> also use <repo>.git; --lfs uses a mirror clone plus the LFS objects in the same directory.

JSON metadata (json/)

Release assets

Assets are downloaded to a temporary .<file>.part next to their destination and renamed into place only when complete, so an interrupted run never leaves a truncated asset. Each download is checked against the size GitHub reports and, when GitHub provides one, its SHA-256 digest; <file>.sha256 records the checksum (<sha256> <file>). On the next run an asset is skipped only if it is complete: its size equals GitHub's and its content matches the digest (or, when GitHub has none, the sidecar). Otherwise it is downloaded again.

Clones Follow GitHub

A mirror clone is updated with git fetch --all. A branch or tag that is deleted on GitHub stays in the backup, so a deleted branch can still be recovered; a force-pushed branch is overwritten, and the commits it used to point at stay in the repository's object store only until git gc removes them. Pass --prune (config key prune) to delete refs that were deleted on GitHub instead, which makes each clone an exact copy of GitHub's current refs. Snapshot the output directory with a tool such as restic, borg or ZFS if you need the history of force-pushed branches.

One case needs pruning to continue: if a deleted branch foo is replaced by a new branch foo/bar, git cannot hold both refs. The update then prunes once, logs a warning, and carries on, so that repository keeps being backed up (the stale refs are lost in that case).

Design Rationale

• git/ and json/ are siblings — git clones and JSON metadata are clearly separated.
• Owner subdirectory — supports backing up multiple users/orgs into a single --output root.
• Stable paths — file names are deterministic: the same repo always maps to the same path, enabling incremental updates.
• Standard git layout — *.git/ directories are valid bare git repos that any git tool can read directly.

GitHub Enterprise Server (GHES)

github-backup can back up a GitHub Enterprise Server instance by overriding the API base URL and, when it differs, the clone hostname.

Verification status. The project's tests run against mock clients and local fake servers. No real GHES instance was available, so GHES-specific behaviour (older API versions, missing endpoints) is not verified. An endpoint that a GHES version lacks answers 404 and is skipped or reported as a failure like any other missing resource.

Configuration

Set --api-url (or GITHUB_API_URL, or api_url in the config file) to your instance's REST API root. It must be an https:// URL:

github-backup myorg \
  --output /backup \
  --api-url https://github.example.com/api/v3 \
  --org --all

(Pass the token in GITHUB_TOKEN.) Every API request is made relative to this base, and git clones use the clone_url the API returns.

Config File

# config.toml  (token comes from GITHUB_TOKEN)
owner = "my-org"
org = true
output = "/backup"
api_url = "https://github.example.com/api/v3"
all = true

TLS and Private Certificate Authorities

If the instance uses an internal CA, make both the tool and git trust it.

System-wide (works for both):

# Debian / Ubuntu
cp my-ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates
# RHEL / Fedora
cp my-ca.crt /etc/pki/ca-trust/source/anchors/ && update-ca-trust

Or per process: SSL_CERT_FILE=/path/to/ca.pem for github-backup and GIT_SSL_CAINFO=/path/to/ca.pem for the git it starts. The tool uses rustls with the operating system's certificate store; it has no flag to disable certificate verification.

Clone URLs

Clone URLs come from the API response (clone_url, or ssh_url with --prefer-ssh). If your instance advertises a hostname that is not reachable from the backup host, or the git endpoint is behind a different load balancer than the API, replace the hostname:

github-backup myorg \
  --output /backup \
  --api-url https://github-api.example.com/api/v3 \
  --clone-host github-git.example.com \
  --org --repositories

Authentication

GHES accepts the same personal access tokens as github.com; create one at https://<your-ghes-host>/settings/tokens. The scopes per category are in Authentication. For organisation backups the token owner must be able to see the repositories (organisation owner, or explicit access).

GitHub Enterprise Cloud (GHEC)

GHEC uses https://api.github.com, like github.com: no --api-url is needed. If the organisation enforces SAML single sign-on, authorise the token for it in GitHub's token settings, otherwise the API answers 403.

Proxies

HTTPS_PROXY, HTTP_PROXY, ALL_PROXY and NO_PROXY apply to the API calls; git follows its own proxy settings. S3 and mirror-destination API calls do not use a proxy. Details and the SOCKS limitation are in Environment Variables.

Rate Limits

GHES rate limits are configured by the site administrator. The tool waits out 429 responses and 403 responses that carry rate-limit information (Retry-After, X-RateLimit-Remaining: 0 or a "rate limit" message) and retries; see Troubleshooting. To reduce the pressure, lower the concurrency:

github-backup myorg --output /backup --org --all --concurrency 2

Docker Guide

The repository ships a multi-stage Dockerfile. The runtime image is based on alpine:3.23 and contains the github-backup binary (a static musl build), git, git-lfs, openssh-client, the CA bundle and tini. It is not a minimal or distroless image: it has a shell (/bin/sh) and the Alpine package manager. Its size has not been measured for the current Dockerfile; build it and run docker images if it matters to you.

Version note. Multi-architecture images (linux/amd64, linux/arm64) are published to GHCR for every release: latest (most recent stable release), X.Y.Z and X.Y. :latest is the last published release; the entrypoint wrapper, --doctor and the other behaviour described on this page are not in v0.3.2. Until the next release, build the image from the repository (docker build -t github-backup .) and use that tag in the examples below. Pin a version in production instead of :latest.

ghcr.io/tomtom215/github-backup-rust:latest     # latest stable release
ghcr.io/tomtom215/github-backup-rust:X.Y.Z      # an exact release
ghcr.io/tomtom215/github-backup-rust:X.Y        # latest patch of X.Y

Running on Unraid? A Community Applications template is bundled: see unraid/README.md. It uses the same image; the rest of this guide still applies.

Quick Start

# Pull the image (or build it: docker build -t github-backup .)
docker pull ghcr.io/tomtom215/github-backup-rust:latest

# Check the setup first: git, output directory, network, token
docker run --rm -e GITHUB_TOKEN \
  ghcr.io/tomtom215/github-backup-rust:latest \
  octocat --doctor

# Run a backup
docker run --rm -e GITHUB_TOKEN \
  -v "$PWD/backups:/backup" \
  ghcr.io/tomtom215/github-backup-rust:latest \
  octocat --output /backup --all

-e GITHUB_TOKEN without a value forwards the variable from your shell; the token then never appears in the docker command line or your shell history (-e GITHUB_TOKEN=$GITHUB_TOKEN would put it into the process list of the host). The container's working directory is /backup, and --output defaults to the working directory, so --output /backup is optional.

The container exit status is the tool's: 0 complete, 3 finished but incomplete (something could not be backed up), 1 could not run, 143 stopped by docker stop. See Exit Codes.

Users and Permissions

The image runs as the numeric user 1000:1000 (backup). A second user, 99:100 (nobody:users, the owner of Unraid shares), also has an account in the image because ssh refuses to run for a UID without one. The /backup volume is owned by 1000:1000.

• A bind mount must be writable by the user the container runs as. Either chown 1000:1000 the host directory, or run as the directory's owner: docker run --user "$(id -u):$(id -g)" ... (Compose: BACKUP_UID and BACKUP_GID in .env).
• Files get the mode implied by umask 022 by default. Set -e UMASK=000 (octal, three or four digits) if other users must write into the share.
• git refuses repositories owned by another user ("dubious ownership"). The tool trusts exactly the path it works on, and the image trusts /backup and /backup/* system-wide, so a volume that was populated by another UID still updates. With another --output path add it: -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='/other/path/*'.
• On Kubernetes set securityContext.runAsUser: 1000 and fsGroup: 1000, otherwise most volume drivers hand you a directory the user cannot write.

How Arguments Reach the Binary

The entrypoint is tini followed by a small POSIX shell wrapper (docker/entrypoint.sh):

• Arguments given (docker run IMAGE octocat --all, Compose command:, Kubernetes args:): passed to github-backup unchanged.
• No arguments: the wrapper builds them from GITHUB_OWNER (the owner), BACKUP_MODE (one of --all, --doctor, --check, --list-scopes, --verify, --tui, --print-config-template, or another flag starting with --) and BACKUP_FLAGS (more flags, split on spaces: quotes are not interpreted, so write --include-repos rust-* and not --include-repos 'rust-*'). This is the Unraid path.
• No arguments and none of those variables: github-backup --help.
• The wrapper refuses ;, backticks, $ and parentheses in BACKUP_FLAGS and shell metacharacters in BACKUP_MODE, and a malformed UMASK, with exit status 64.
• A variable that is set but empty (VAR=, which Compose and Unraid pass for every blank field) is removed for these: GITHUB_TOKEN, GITHUB_API_URL, GITHUB_CLONE_HOST, GITHUB_OAUTH_CLIENT_ID, MIRROR_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, BACKUP_ENCRYPT_KEY, BACKUP_NOTIFY_WEBHOOK, GITHUB_BACKUP_RESTORE_YES and the proxy variables.

Docker Compose

docker-compose.yml defines one service per scenario; the non-default ones sit behind --profile. Copy the template once:

cp compose.example.env .env
chmod 600 .env
$EDITOR .env            # at least GITHUB_TOKEN

Fixed flags of a profile (--s3-bucket ...) are part of the service's entrypoint:, so whatever you add after the service name is appended:

Profile matrix

docker compose run --rm backup octocat --all                          # local
docker compose --profile doctor run --rm doctor octocat              # pre-flight
docker compose --profile tui run --rm tui octocat                    # TUI
docker compose --profile verify run --rm verify octocat              # verify
docker compose --profile s3 run --rm backup-s3 octocat --all         # S3
docker compose --profile b2 run --rm backup-b2 octocat --all         # B2
docker compose --profile minio run --rm backup-minio octocat --all   # MinIO
docker compose --profile codeberg run --rm backup-codeberg octocat --all
docker compose --profile gitlab run --rm backup-gitlab octocat --all

Notes:

• The compose file does not bundle an S3 server (MINIO_ENDPOINT points at one you run, with the bucket created beforehand; the tool never creates buckets). Do not publish a MinIO server's ports on all interfaces with default credentials.

• Only the JSON metadata goes to S3, and only metadata reaches B2/MinIO; the clones stay in ./backups/. To copy the code off-site use the mirror profiles or a file-level tool.

• Compose does not forward HTTPS_PROXY; add HTTPS_PROXY: to the x-env block if the container needs a proxy (a host-side 127.0.0.1 proxy is not reachable from the container).

• A config file is not mounted by default; mount it for the run:

  docker compose run --rm -v "$PWD/config.toml:/etc/github-backup/config.toml:ro" \
    backup --config /etc/github-backup/config.toml
  

Environment Variables

Everything in compose.example.env is also accepted by a plain docker run:

Compose forwards only the variables of the x-env block of docker-compose.yml and the credentials of each profile; add any other variable (for example AWS_SESSION_TOKEN or a proxy) to that block.

Security Notes

• Non-root: the image runs as 1000:1000 (numeric, so Kubernetes runAsNonRoot: true can verify it).
• A shell is present: this is an Alpine image with apk, sh and ssh. Nothing is removed from it.
• Secrets: pass them as environment variables (-e GITHUB_TOKEN, --env-file, Compose .env, Kubernetes secrets), never baked into the image or on a command line. The tool hands the token to git through an environment variable and a host-scoped credential helper; it is not written to a file and not placed in any argument list.
• SSH: --prefer-ssh needs keys and a known_hosts file; the image ships none. Mount them and point GIT_SSH_COMMAND at them.
• tini as PID 1 forwards SIGTERM (docker stop, pod eviction) to the backup, which stops its git processes, releases its locks and exits with 143 within a few seconds, well inside Docker's default 10 s grace period. The checkpoint it leaves lets the next run resume (within 6 hours).
• Atomic outputs: reports, metrics and JSON files are written to a temporary name and renamed, so a monitor never reads a half-written file.
• Not covered: the image is not signed or attested (only the release binaries are), and it has not been scanned by the project.

Scheduled Backups

The image is a one-shot tool: it runs to completion and exits. Use a scheduler that starts a new container per run.

Plain cron

# /etc/cron.d/github-backup   (the token comes from a 0600 env file)
0 2 * * * backup docker run --rm --env-file /etc/github-backup/env \
  -v /var/backup/github:/backup \
  ghcr.io/tomtom215/github-backup-rust:X.Y.Z \
  octocat --output /backup --all >> /var/log/github-backup.log 2>&1

systemd timer

# /etc/systemd/system/github-backup.timer
[Unit]
Description=Daily GitHub backup

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15m

[Install]
WantedBy=timers.target

# /etc/systemd/system/github-backup.service
[Unit]
Description=GitHub backup
After=network-online.target docker.service
Wants=network-online.target
Requires=docker.service
OnFailure=notify-github-backup-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/bin/docker run --rm --env-file /etc/github-backup/env -v /var/backup/github:/backup ghcr.io/tomtom215/github-backup-rust:X.Y.Z octocat --output /backup --all

Keep the secret in /etc/github-backup/env (mode 0600, one NAME=value per line). systemd does not expand shell \ continuations in ExecStart=, so keep that line whole. A run that exits 3 marks the unit failed and triggers OnFailure=.

Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: github-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid          # one backup at a time
  failedJobsHistoryLimit: 3
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2                # a failed run is retried at most twice
      template:
        spec:
          restartPolicy: Never
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
            fsGroup: 1000
            runAsNonRoot: true
          containers:
            - name: backup
              image: ghcr.io/tomtom215/github-backup-rust:X.Y.Z
              args: ["octocat", "--output", "/backup", "--all"]
              env:
                - name: GITHUB_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: github-backup-token
                      key: token
              volumeMounts:
                - name: backup
                  mountPath: /backup
              resources:
                requests: { cpu: "100m", memory: "128Mi" }
                limits: { cpu: "1", memory: "512Mi" }
          volumes:
            - name: backup
              persistentVolumeClaim:
                claimName: github-backup-pvc

A job whose backup exits 3 (incomplete) counts as failed and is retried up to backoffLimit times; each retry resumes and retries only what failed.

Troubleshooting

# What the doctor sees inside the container
docker compose --profile doctor run --rm doctor octocat

# Print the OAuth scopes a flag set needs (note: it does not expand --all)
docker run --rm -e GITHUB_TOKEN ghcr.io/tomtom215/github-backup-rust:latest \
  octocat --org --repositories --issues --list-scopes

# Validate a config file and connectivity without running a backup
docker run --rm -e GITHUB_TOKEN \
  -v "$PWD/config.toml:/etc/github-backup/config.toml:ro" \
  ghcr.io/tomtom215/github-backup-rust:latest \
  --config /etc/github-backup/config.toml --check

More: Troubleshooting and the Operations Runbook.

Systemd Timer

Run github-backup automatically on a schedule using a systemd service + timer pair.

Setup

1. Create a dedicated user

sudo useradd -r -m -d /var/backup/github -s /sbin/nologin github-backup
sudo mkdir -p /var/backup/github
sudo chown github-backup:github-backup /var/backup/github

2. Store the token securely

sudo mkdir -p /etc/github-backup
sudo tee /etc/github-backup/secrets.env > /dev/null <<'EOF'
GITHUB_TOKEN=ghp_your_token_here
EOF
sudo chmod 600 /etc/github-backup/secrets.env
sudo chown root:github-backup /etc/github-backup/secrets.env

3. Create a config file

sudo tee /etc/github-backup/config.toml > /dev/null <<'EOF'
owner = "octocat"
output = "/var/backup/github"
concurrency = 8
repositories = true
issues = true
pulls = true
releases = true
wikis = true
gists = true
EOF
sudo chmod 644 /etc/github-backup/config.toml

4. Create the service unit

sudo tee /etc/systemd/system/github-backup.service > /dev/null <<'EOF'
[Unit]
Description=GitHub Backup
After=network-online.target
Wants=network-online.target
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
User=github-backup
Group=github-backup
EnvironmentFile=/etc/github-backup/secrets.env
ExecStart=/usr/local/bin/github-backup --config /etc/github-backup/config.toml --report /var/lib/github-backup/report.json
StateDirectory=github-backup
StandardOutput=journal
StandardError=journal
SyslogIdentifier=github-backup

# Hardening
ProtectSystem=strict
ReadWritePaths=/var/backup/github
PrivateTmp=true
NoNewPrivileges=true
EOF

5. Create the timer unit

sudo tee /etc/systemd/system/github-backup.timer > /dev/null <<'EOF'
[Unit]
Description=Run GitHub Backup daily at 02:00
Requires=github-backup.service

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target
EOF

6. Enable and start

sudo systemctl daemon-reload
sudo systemctl enable --now github-backup.timer

7. Verify

# Check timer status
systemctl status github-backup.timer

# Run immediately to test
sudo systemctl start github-backup.service

# Follow logs
journalctl -u github-backup.service -f

Exit Status and Failure Handling

The unit fails whenever the exit status is not 0: 1 (the run could not run), 3 (the run finished but something could not be backed up) and 130 / 143 (stopped). OnFailure= therefore fires for an incomplete backup too, and systemctl status github-backup.service shows the status. Do not add SuccessExitStatus=3: that would hide incomplete backups. journalctl shows the summary banner with the list of failures. See Monitoring for the report file, metrics and webhook.

A failure unit that sends mail:

# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Notify on failure of %i

[Service]
Type=oneshot
ExecStart=/usr/bin/mail -s "%i failed on %H" ops@example.com

StateDirectory=github-backup creates /var/lib/github-backup (owned by the service user) for the --report file; with ProtectSystem=strict the report path must be writable, which StateDirectory= provides.

Stopping and Timeouts

systemctl stop sends SIGTERM. The backup stops its git processes, releases its lock and exits 143 within a few seconds; the next run resumes (within 6 hours) from the checkpoint. No cleanup of lock files is ever needed.

Multiple Owners

Create separate service/timer pairs per owner, or one script that does not stop at the first failure:

#!/bin/bash
# /usr/local/bin/github-backup-all.sh
status=0
for owner in octocat myorg another-org; do
  github-backup "$owner" --config /etc/github-backup/config.toml \
    --output /var/backup/github || status=$?
done
exit "$status"

(Each owner needs its own --report and --prometheus-metrics file if you use them: both hold one run. If the config file has org = true it applies to every owner; keep organisations and users in separate configs.) The last non-zero status is the one systemd sees.

Monitoring

systemctl list-timers github-backup.timer
journalctl -u github-backup.service --since "yesterday"

Cron

Schedule github-backup with cron for periodic backups. (A systemd timer gives better logging and failure handling when it is available.)

Keep the Token Out of the Crontab

Put the token in a file readable only by the backup user and load it in a small wrapper:

sudo install -d -m 750 -o github-backup -g github-backup /etc/github-backup
sudo -u github-backup sh -c 'umask 077; echo "GITHUB_TOKEN=ghp_your_token_here" > /etc/github-backup/secrets.env'

#!/bin/bash
# /etc/github-backup/run.sh   (chmod 750)
set -uo pipefail
set -a; . /etc/github-backup/secrets.env; set +a
LOG=/var/log/github-backup.log

/usr/local/bin/github-backup --config /etc/github-backup/config.toml \
    --report /var/lib/github-backup/report.json >> "$LOG" 2>&1
status=$?

# 0 = complete, 3 = finished but incomplete, 1 = could not run, 130/143 = stopped.
# Whatever this script prints is mailed by cron, so print only on failure.
if [ "$status" -ne 0 ]; then
  echo "github-backup exited with status $status on $(hostname); last log lines:" >&2
  tail -n 20 "$LOG" >&2
fi
exit "$status"

Crontab for the github-backup user:

# Daily at 02:00.  The wrapper logs everything to /var/log/github-backup.log and
# prints (so cron mails) only when the backup failed or is incomplete.
MAILTO=ops@example.com
0 2 * * * /etc/github-backup/run.sh

Cron mails whatever a job prints, and github-backup writes its log to stderr, so a job without the wrapper's redirect would mail the whole log every day. Test the wrapper by hand (sudo -u github-backup /etc/github-backup/run.sh) before relying on it: cron runs with a minimal environment, so use absolute paths. The log file and the report directory must be writable by the backup user (install -d -o github-backup /var/lib/github-backup, and create /var/log/github-backup.log with touch and chown once).

Failure Detection

The exit status is the signal: 0 complete, 3 finished but incomplete, 1 could not run. Do not assume "no error mail" means success. For stronger monitoring use --prometheus-metrics, --notify-webhook or an external check of the report file's age; see Monitoring.

Log Rotation

/var/log/github-backup.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 github-backup adm
}

Save it as /etc/logrotate.d/github-backup.

Notes

• Do not run two backups for the same owner at once: the second one stops with "another backup ... is already running" (exit 1). A run that outlasts the schedule interval is therefore harmless, but it shows up as a failed run.
• The user running cron needs write access to --output and to the --report directory.
• Prefer systemd timers where available.

Monitoring & Reporting

A backup is only as good as your knowledge that it ran. github-backup reports every run in six ways, all derived from one list of failures so that they cannot disagree:

A --dry-run produces none of them except the log and the banner.

Exit Status

Any non-zero status means "do not trust this run". 3 is the interesting one: the other repositories were backed up, and re-running retries only what failed. Failures are isolated per repository and per category, so a repository whose issue list cannot be fetched is still cloned, and one broken repository never stops the others. Full table: CLI Reference.

JSON Summary Report

--report FILE writes a JSON file at the end of every run, including a run that failed early (so a monitor never keeps reading the previous run's "success"). The file is replaced atomically.

github-backup octocat --output /var/backup/github --all \
  --report /var/log/github-backup/report.json

Schema

{
  "tool_version": "X.Y.Z",
  "schema_version": 1,
  "owner": "octocat",
  "started_at": "2026-01-15T04:00:00Z",
  "finished_at": "2026-01-15T04:02:23Z",
  "duration_secs": 142.7,
  "repos_discovered": 42,
  "repos_backed_up": 41,
  "repos_skipped": 0,
  "repos_errored": 1,
  "gists_backed_up": 5,
  "issues_fetched": 1204,
  "prs_fetched": 387,
  "workflows_fetched": 18,
  "failure_count": 1,
  "failures": [
    {
      "scope": "octocat/big-repo",
      "step": "repository",
      "message": "git clone ... failed (exit 128): ..."
    }
  ],
  "success": false
}

(tool_version is the version of the binary that wrote the file; X.Y.Z stands for it here.)

failures includes things that are not repositories: an S3 upload that failed (scope: "post-processing", step: "s3 sync"), a mirror push that was refused ("mirror push"), a manifest that could not be written, a starred repository that could not be cloned. message text is scrubbed of the token.

Shell wrapper

#!/usr/bin/env bash
set -uo pipefail
REPORT=/var/log/github-backup/report.json

github-backup octocat --output /backup --all --report "$REPORT"
status=$?

if [ "$status" -ne 0 ]; then
  mail -s "github-backup exit $status on $(hostname)" ops@example.com < "$REPORT"
fi
exit "$status"

Testing the exit status is enough; the report adds the reason. With jq:

jq -r '.failures[] | "\(.scope): \(.step)"' /var/log/github-backup/report.json

Webhook

--notify-webhook URL (or BACKUP_NOTIFY_WEBHOOK) sends one JSON POST after the run, also when the run failed:

{
  "status": "partial",
  "owner": "octocat",
  "timestamp": "2026-01-15T04:02:23Z",
  "repos_backed_up": 41,
  "repos_errored": 1,
  "failure_count": 1,
  "failed": [ { "scope": "octocat/big-repo", "step": "repository" } ]
}

failed lists at most the first 20 failures and never includes messages (they can be long and the receiver is an external service); failure_count is the full number. A webhook that cannot be reached, or answers an error, is logged as a warning and does not change the exit status. The notification is not sent for a dry run, nor when the tool stops before the run starts (a bad argument, an unreadable config file, a lock held by another run, a failed device-flow login): treat the exit status as the primary signal and the webhook as a convenience.

Use an https:// URL: the payload names the owner and a plain-http:// URL produces a warning. The log shows only the scheme and host of the URL, never its path or query. The request honours HTTPS_PROXY.

Prometheus Metrics

--prometheus-metrics FILE writes metrics in the Prometheus text format for node_exporter's textfile collector. The file is replaced atomically after every run, including failed ones.

github-backup octocat --output /backup --all \
  --prometheus-metrics /var/lib/node_exporter/textfile_collector/github_backup.prom

All metrics carry an owner label. The file holds one owner: when you back up several owners, give each its own file.

github_backup_last_success_timestamp_seconds is carried over from the previous file when a run fails, so it keeps pointing at the last good run. If no run has ever succeeded the metric is absent.

Alert rules:

groups:
  - name: github-backup
    rules:
      - alert: GitHubBackupFailed
        expr: github_backup_success == 0
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "github-backup for {{ $labels.owner }} finished with failures"
      - alert: GitHubBackupStale
        expr: time() - github_backup_last_success_timestamp_seconds > 2 * 24 * 3600
        labels: { severity: critical }
        annotations:
          summary: "No complete github-backup for {{ $labels.owner }} in two days"
      - alert: GitHubBackupNeverSucceeded
        expr: absent(github_backup_last_success_timestamp_seconds)
        for: 1d
        labels: { severity: warning }

If the tool does not run at all (timer stopped, host down) no new file appears; GitHubBackupStale catches that, GitHubBackupFailed does not. Without node_exporter, push the same values from the JSON report to a Pushgateway.

Alerting With systemd

A failed run exits non-zero, so systemd marks the unit failed and OnFailure= fires for exit status 1 and 3:

# /etc/systemd/system/github-backup.service
[Unit]
Description=GitHub Backup
OnFailure=notify-github-backup-failure@%n.service

[Service]
Type=oneshot
EnvironmentFile=/etc/github-backup/secrets.env
ExecStart=/usr/local/bin/github-backup \
  --config /etc/github-backup/config.toml \
  --report /var/log/github-backup/report.json

# /etc/systemd/system/notify-github-backup-failure@.service
[Unit]
Description=Notify on failure of %i

[Service]
Type=oneshot
ExecStart=/usr/bin/mail -s "%i failed on %H" ops@example.com

See Systemd Timer for the full setup. With cron, set MAILTO (cron mails any output) or wrap the command as in the shell wrapper above.

Logs

Logs go to stderr, one event per line (-v adds debug detail, -q shows errors only). Do not build alerts on matching WARN or ERROR text: both levels occur during a run that ends normally. What the lines look like:

INFO repository processed repo=octocat/hello progress="2/3"
WARN step failed, continuing with the rest scope=octocat/big-repo step=repository error=git clone ...
INFO backup finished owner="octocat" stats=repos: 2/3 backed up, 0 skipped, 1 errored; gists: 0 backed up; issues: 5 fetched; PRs: 2 fetched; workflows: 1 fetched; 1 failure(s) (14.3s elapsed)
ERROR backup is incomplete: 1 item(s) could not be backed up (exit status 3) failures=1

step failed, continuing (WARN) is one recorded failure; fatal error: stopping the run (ERROR) ends the run early (rejected credentials, exhausted rate-limit budget, full disk); the final backup is incomplete / backup failed line states the outcome. The summary banner printed at the end lists each failure with its reason.

Incremental Runs and the State File

Run github-backup as often as you like: the second run is faster, and it never loses data that the first one captured.

What happens each run:

• The repository list, issues.json and pulls.json are always fetched in full and merged into the stored files (an item that GitHub no longer lists stays in the backup). The files can only grow or change; a run never shrinks them.
• The expensive part, the per-item requests (issue_comments, issue_events, pull_comments, pull_commits, pull_reviews), is skipped for an issue or pull request that has not been updated since the repository's watermark and whose files are already on disk. A missing file is always fetched.
• Git clones are updated with git fetch, never cloned again.

The watermarks live in <output>/<owner>/json/backup_state.json:

• One watermark per repository, set to 15 minutes before the run started (a margin for clock differences and replication lag).
• A repository's watermark only advances when every step for it succeeded. A repository that failed keeps its old watermark, so the next run fetches what the failed run missed.
• A watermark covers the categories that were enabled when it was recorded. Turning on another per-item category later makes the first run fetch everything for it.
• last_successful_run is informational; it only moves after a run without failures.
• A missing or unreadable file means "fetch everything once".

Controls:

Limitation. "Unchanged" is judged by the issue's or pull request's updated_at timestamp. GitHub does not guarantee that every edit to a comment, review or event bumps that timestamp, so an in-place edit of an old comment can be missed by an incremental run. Run with --full now and then (for example weekly) if exact copies matter.

Further Reading

• Systemd Timer and Cron: scheduling
• CLI Reference: every flag and the exit codes
• Operations Runbook: what to do when an alert fires

Restore Guide

This guide covers getting data back: the automated --restore mode (labels, milestones, issues), the manual procedures for everything else, and decrypting S3 objects. Read What Is Not Backed Up before you rely on any of it.

Automated Restore (--restore)

--restore re-creates labels, milestones and issues from the JSON backup in the repositories of a target organisation (or user) through the GitHub REST API.

It is a mode of its own:

• It reads the local backup under --output; it makes no backup first and does not contact the source account, so it works when the source repository or account has been deleted.
• The target repositories must already exist, with the same names as in the backup. Restore never creates repositories; create them first (code with the git procedure below).
• It asks for confirmation before it does anything.

Basic usage

export GITHUB_TOKEN=ghp_dummy_write_token     # a token with write access
github-backup octocat \
  --output /var/backup/github \
  --restore \
  --restore-target-org new-org

OWNER (octocat here) names the backup to read; --restore-target-org names where to write (default: the same name as OWNER; do not restore into the repositories you backed up). The tool prints a warning banner and asks you to type yes. When stdin is not a terminal, confirm with the flag or the environment variable:

github-backup octocat --output /var/backup/github \
  --restore --restore-target-org new-org --restore-yes
# or
GITHUB_BACKUP_RESTORE_YES=1 github-backup octocat --output ... --restore ...

Dry run

--dry-run shows what would be created without asking for confirmation and without any API call:

github-backup octocat --output /var/backup/github \
  --restore --restore-target-org new-org --dry-run

What is restored

Details and limits:

• Issues are created with their title, body and labels. The body gets a footer saying where it came from (Restored by github-backup from octocat/repo#12 (opened by @alice on 2024-03-01T...)) because GitHub sets the author and date to the restoring token and the time of restore. Closed issues come back open; assignees, milestone links, comments, reactions and timeline events are not restored. Issue numbers are not preserved.
• Pull requests (entries of issues.json that carry a pull_request property) are skipped; GitHub has no API to import them.
• Milestones keep title, description, state and due date; labels keep name, colour and description.
• Within each repository the order is labels, then milestones, then issues, so the labels exist when the issues that use them are created.

Repeating a restore

Labels and milestones that already exist (HTTP 422) are counted as skipped. Every restored issue carries a hidden marker naming its source (<!-- github-backup-restore:octocat/repo#12 -->); before restoring a repository's issues the tool lists the target's issues (open and closed) and skips any whose marker it finds. So running the same restore twice does not duplicate anything. The marker lives in the issue body: an issue whose body was edited to remove it, or that was deleted, is created again.

Outcome and exit status

A repository that cannot be restored (target missing, no write access) is logged and counted; the others continue. The final line shows labels: N created, N skipped, N errored | milestones: ... | issues: .... Exit status: 0 all fine, 3 some items could not be restored, 1 the restore could not start (no backup found, confirmation refused).

Token requirements

The fine-grained row is taken from GitHub's documentation and has not been exercised against a live account by the project.

What Is Not Restored, or Not Backed Up

Be clear about what a restore cannot give back:

• Pull requests, issue and PR comments, reviews, reactions and timeline events: saved in the JSON files, but there is no importer.
• Discussions and Projects: not backed up at all (GitHub's REST API has no endpoints for them; --discussions / --projects do nothing).
• Issue attachments and images: the JSON holds the links; the files are not downloaded.
• Actions secrets and variables, code-scanning and Dependabot alerts, rulesets, team permissions, commit comments and statuses, wiki attachments, stargazers of your repositories: not collected.
• Git LFS objects: only with --lfs (needs git-lfs); without it the clone holds LFS pointers only.
• Starred gists: metadata only (no content).
• Repository clones in S3: S3 receives the JSON only. If the local backup is lost, the git data is gone unless you have a mirror (--mirror-to) or a copy of <output>/<owner>/git/.
• Everything needs a token that could see it in the first place: private data of other users or organisations is never visible.

Git Data

The clones under git/repos/<repo>.git are normal bare repositories. The manual procedures in the rest of this page use standard git and GitHub behaviour; the project has not run them against github.com.

Do not use git push --mirror to push to GitHub. A mirror clone of a GitHub repository contains refs/pull/*, which GitHub does not accept ("deny updating a hidden ref"), so the push fails. Push branches and tags:

# 1. create the empty target repository first (the push does not create it)
gh repo create new-org/my-repo --private

# 2. push branches and tags
git -C /backup/octocat/git/repos/my-repo.git push --prune \
    https://github.com/new-org/my-repo.git \
    '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'

--prune deletes branches and tags at the destination that are not in the backup; leave it out when pushing into a non-empty repository. Pull request refs and the pull requests themselves do not come back.

Wikis

The wiki repository only exists once the wiki has a first page (create one in the web UI), then:

git -C /backup/octocat/git/wikis/my-repo.wiki.git push \
    https://github.com/new-org/my-repo.wiki.git '+refs/heads/*:refs/heads/*'

Gists

Gist git data lives in git/gists/<gist-id>.git/; the description and visibility are in json/gists/<gist-id>.json. Create a new gist in the web UI (or with gh gist create), then push the content:

git -C /backup/octocat/git/gists/abc123.git push --force \
    https://gist.github.com/<new-gist-id>.git 'refs/heads/*:refs/heads/*'

Releases and Assets

Assets are under json/repos/<repo>/release_assets/<tag>/<file>:

gh release create v1.0.0 \
  /backup/octocat/json/repos/my-repo/release_assets/v1.0.0/my-binary \
  --title "v1.0.0" --notes "Restored from backup" \
  --repo new-org/my-repo

releases.json has the release notes, draft and prerelease flags.

Labels via curl

If you prefer scripting over --restore:

jq -c '.[]' /backup/octocat/json/repos/my-repo/labels.json | \
while read -r label; do
  name=$(jq -r '.name' <<< "$label")
  color=$(jq -r '.color' <<< "$label")
  desc=$(jq -r '.description // ""' <<< "$label")
  curl -s -X POST \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg n "$name" --arg c "$color" --arg d "$desc" '{name:$n,color:$c,description:$d}')" \
    "https://api.github.com/repos/new-org/my-repo/labels"
done

Branch Protection, Deploy Keys, Collaborators

branch_protections.json (an object keyed by branch name), deploy_keys.json and collaborators.json are records. There is no automated path back: the GitHub endpoint that sets branch protection (PUT .../branches/<b>/protection) expects a different request shape than the response that was saved, so the rules have to be translated by hand. Deploy keys can be re-added from the public key material; collaborators have to be re-invited.

Starred, Followed and Organisation Data

These files are reference archives (useful for auditing or rebuilding the social graph by hand); nothing restores them automatically.

Decrypting S3 Objects

If you used --encrypt-key, objects in the bucket end in .enc. Download one with your provider's tool, then:

export BACKUP_ENCRYPT_KEY=...        # the 64-hex-character key used for the upload
github-backup --decrypt \
  --decrypt-input issues.json.enc --decrypt-output issues.json

--decrypt needs no OWNER and no network. The wire format is [12-byte random nonce][ciphertext + 16-byte GCM tag]. openssl enc cannot decrypt it (it does not support AES-GCM); use the tool, or any AES-GCM library, as shown in At-Rest Encryption.

Layout Reference

Where each file lives is on one page: Output Directory Layout. The implementation of --restore is described in Restore Implementation.

Operations Runbook

Day-to-day procedures for a production github-backup deployment: health checks, what to do when a run fails or is interrupted, key rotation, verification and upgrades. How the signals are produced is described in Monitoring & Reporting.

Daily Health Check

After each scheduled run, confirm:

1. Exit status. 0 means complete. 3 means the run finished but is incomplete (some items failed). 1 means it could not be carried out. 130 / 143 mean it was interrupted. Anything but 0 needs a look.

2. Failures (from the report, if --report is configured):

   jq '{success, failure_count, repos_errored}' /var/log/github-backup/report.json
   jq -r '.failures[] | "\(.scope): \(.step): \(.message)"' /var/log/github-backup/report.json
   

   The same list is printed at the end of the run in the summary banner.

3. Prometheus metrics (if --prometheus-metrics is configured):

   grep -E 'github_backup_(success|failures|last_success)' /var/lib/node_exporter/textfile_collector/github_backup.prom
   # github_backup_success{owner="..."} 1
   

4. History (always written): the last runs, newest last:

   jq '.entries[-3:]' /var/backup/github/octocat/json/backup_history.json
   

5. S3 (if configured): a failed upload or deletion is a recorded failure (step: "s3 sync") and makes the run exit 3; the log line S3 sync complete uploaded=N skipped=M errored=0 deleted=0 shows the counts.

Backup Interrupted

SIGINT, SIGTERM (exit 130 / 143)

Ctrl+C, systemctl stop, docker stop and a Kubernetes eviction send these. The tool stops running git (the whole process group is killed), abandons in-flight API requests, releases the lock and exits within about a second or a few; nothing is left half-written under its real name. What to expect:

1. <output>/<owner>/json/backup_checkpoint.json lists the repositories that were finished.
2. Re-run the same command. Within 6 hours of the interruption the run resumes and skips those repositories; after that the checkpoint is ignored and every repository is refreshed. Either way the git clones are only updated, not cloned again, and a repository that was in the middle of its first clone starts that clone again (the partial one is in a hidden staging directory, removed automatically).
3. backup_state.json is not advanced by an interrupted run, so nothing is skipped wrongly next time.

SIGKILL, out-of-memory kill, power loss

The operating-system lock is released by the kernel, so no stale lock exists and nothing has to be deleted before the next run (the lock files stay on disk; they are only markers). A git process may keep running for a while after its parent was killed (it is not told to stop); it ends on its own, and the next run removes leftover staging directories. Start the next run normally.

Investigating Failures

Exit status 3 or success: false means one or more steps failed while the rest completed. Each entry of failures names a scope (a repository, the owner or post-processing), the step and the error text.

Re-running retries what failed and keeps everything that succeeded.

Rate limit

The client waits out 429 responses and 403 responses that carry rate-limit information (Retry-After, X-RateLimit-Remaining: 0 or a "rate limit" message) and retries (up to 6 times, at most about an hour of waiting per request). The log says rate limited; waiting Ns. If the wait would exceed the budget the run stops with a rate limit exceeded error (exit 1) and the next run starts from the checkpoint. Check the quota:

curl -s -H "Authorization: Bearer $GITHUB_TOKEN" https://api.github.com/rate_limit | jq '.rate'

A token has 5 000 requests per hour (classic or fine-grained; GitHub App and OAuth app tokens of an Enterprise Cloud organisation get more). Reduce the load with --concurrency 1, fewer categories, or --include-repos slices spread over several runs. Issue and pull request lists cost one request per 100 items every run; the per-item requests are what the watermarks save.

S3 failures

A failed upload is a recorded failure; the message carries the HTTP status, the S3 error code and a hint. Common causes:

• SignatureDoesNotMatch / InvalidAccessKeyId: wrong or expired credentials, or an endpoint belonging to another provider.
• NoSuchBucket: the bucket does not exist (it is never created) or the endpoint or region is wrong.
• AccessDenied: the policy lacks s3:ListBucket, s3:GetObject or s3:PutObject (see required permissions).
• RequestTimeTooSkewed: fix the system clock.
• "cannot reach the endpoint": network, DNS or a proxy (S3 ignores HTTPS_PROXY).

--dry-run does not test S3 (it skips the sync).

Retention and Deleted Data

--keep-last and --max-age-days are deprecated and ignored: the tool keeps one continuously updated backup per owner and never deletes snapshot directories. If you need point-in-time copies, snapshot the output directory with restic, borg, ZFS or LVM, and apply the retention policy there.

Know what "update" means for deletions: by default a mirror update keeps branches and tags that were deleted on GitHub, but it follows force-pushes (the old commits of a force-pushed branch are not kept, and become unreachable until git gc removes them). Add --prune to delete refs that were deleted on GitHub, and keep snapshots if you must be able to recover an earlier state. issues.json and pulls.json keep items that disappear from GitHub; other lists mirror the current state.

S3 stale object cleanup

With --s3-delete-stale objects under <prefix>/<owner>/json/ whose local file is gone are deleted, but never when the run had failures, when the local tree could not be fully read or is empty, or when an upload failed. Deletion is permanent unless the bucket is versioned. There is no preview (a --dry-run skips the S3 step).

Encryption Key Rotation

Changing the key makes the next run upload every file again under the new key (the stored content digest is keyed). Objects whose local file is gone stay under the old key. The full procedure, including what to keep and when to retire the old key, is in the encryption guide.

Short form:

1. Generate a new key: openssl rand -hex 32.
2. Run a backup with BACKUP_ENCRYPT_KEY=<new key>.
3. Decrypt one object with the new key to verify it.
4. Optionally run once with --s3-delete-stale to drop objects still under the old key, then retire the old key.

Do not run aws s3 rm ... --include "*.enc": it deletes every encrypted object, new ones included.

Verifying Backup Integrity

If --manifest was used, verify at any time (no network):

github-backup octocat --output /var/backup/github --verify

This compares the SHA-256 of every data file under json/ (not the history, state, checkpoint and lock files) with json/backup_manifest.json and exits 1 if a file is missing, changed or unexpected. It does not examine the git clones: check those with git -C <repo>.git fsck. The manifest protects against accidental damage and casual edits only: it is stored next to the files it describes and is not signed, so someone who can write the directory can rewrite both.

Restoring After Disaster

See the Restore Guide for the full procedure and its limits. Quick reference:

# 1. Git data: create the empty repository first, then push branches and tags
git -C /backup/octocat/git/repos/my-repo.git push --prune \
    https://github.com/new-org/my-repo.git \
    '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'

# 2. Labels, milestones and issues (from the local backup, no backup run first)
GITHUB_TOKEN=ghp_dummy_write_token \
github-backup octocat --output /backup --restore --restore-target-org new-org --restore-yes

Upgrade Procedure

1. Read the changelog, in particular the Unreleased / new version entry (breaking changes are listed there).
2. Stop the scheduled backup (systemd timer or cron job); wait for a running backup to finish.
3. Replace the binary (or pull the new image) and check github-backup --version.
4. Check the setup: github-backup octocat --doctor, and try the real configuration without writing anything: github-backup --config ... --dry-run.
5. Run one backup manually and check the exit status.
6. Resume the schedule.

The state, history and checkpoint files are read by newer versions; a file from an older version only costs a full fetch. An older binary does not understand config keys added by a newer one (unknown keys are rejected).

Log Levels

Set RUST_LOG=github_backup=debug for fine-grained control; RUST_LOG overrides -v and -q. No level prints the token.

Architecture

github-backup is a Rust workspace of seven crates. This page describes how they fit together. (It is also the book page Development / Architecture.)

Workspace Layout

github-backup-rust/
├── crates/
│   ├── github-backup-types/    # GitHub API models, Raw<T>/Page<T>, configuration types
│   ├── github-backup-client/   # Async HTTP client (GitHub API, OAuth, proxy support)
│   ├── github-backup-core/     # Backup engine: orchestration, storage, git, locks
│   ├── github-backup-mirror/   # Push mirrors to Gitea / Forgejo / Codeberg / GitLab
│   ├── github-backup-s3/       # S3-compatible client, sync, AES-256-GCM encryption
│   ├── github-backup-tui/      # Ratatui front-end (--tui)
│   └── github-backup/          # CLI binary (clap), run modes, reporting
├── docker/entrypoint.sh        # container entrypoint wrapper
├── Dockerfile, docker-compose.yml
├── unraid/                     # Unraid Community Applications template
├── docs/                       # mdBook sources
└── deny.toml                   # cargo-deny: licences, bans, advisories, sources

Crate Responsibilities

github-backup-types

Pure data, no I/O.

• GitHub models (Repository, Issue, PullRequest, ...). Fields GitHub can send as null are Options.
• Raw<T> pairs a typed view with the original JSON object, and Page<T> is a list of them. The engine writes the original objects (all properties, in GitHub's key order; serde_json is built with preserve_order), so the backup is lossless; the typed view is only used to make decisions. An element the typed model cannot read is still written verbatim and reported in the log.
• Configuration: BackupOptions, CloneType, ConfigFile (the TOML schema, with deny_unknown_fields), OutputConfig (all output paths), the persisted BackupState, BackupCheckpoint, BackupRunHistory and the starred queue.
• glob_match() for --include-repos / --exclude-repos.

github-backup-client

Async client built on hyper + rustls (no OpenSSL, no reqwest).

• GitHubClient (standard GitHub and, with with_api_url, GHES) behind the object-safe BackupClient trait, which tests replace with MockClient.
• Pagination with Link headers; wrapped responses ({"workflows": [...]}) are merged into plain lists.
• Retry policy (execute_with_retry): rate limits (429, or 403 with Retry-After, X-RateLimit-Remaining: 0 or a rate-limit message) are waited out and retried (up to 6 times, about an hour in total per request); GET requests retry 5xx and transport failures 3 times with jittered back-off; other statuses fail at once. Every response body is capped at 16 MiB and every request has a 120 s timeout.
• Own-account listings: GET /user is asked once; when the login equals the owner the client lists /user/repos?affiliation=owner&visibility=all and /gists, which include private repositories and secret gists.
• Release assets are streamed to disk through an AssetSink; the Authorization header is dropped on any redirect that leaves the origin.
• proxy: HTTP CONNECT proxy support from HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, NO_PROXY (no SOCKS), shared with the webhook and --doctor.
• oauth: the device flow.

github-backup-core

The engine and its abstractions.

BackupEngine<C: BackupClient, S: Storage, G: GitRunner>
  ├── C  (API calls)
  ├── S  (write JSON / bytes atomically; production: FsStorage)
  └── G  (git subprocesses; production: ProcessGitRunner)

• Failure model (engine/steps.rs): every unit of work is a step. A failing step is recorded in BackupStats (scope, step, message) and the run carries on, so one bad repository or category never costs the rest. run returns Err only for things that make continuing pointless (CoreError::is_fatal: rejected credentials, exhausted rate-limit budget, full disk, cancellation) or when the repository list cannot be fetched. Callers read BackupStats::failures() and decide the exit status.
• Run order: owner-level data, packages, starred clones, gists, then the repository list (written to repos.json), then repositories concurrently (bounded by --concurrency); per repository the steps are repository clone, wiki, issues, pull requests, releases, labels, milestones, hooks, security advisories, topics, branches, deploy keys, collaborators, actions, environments (plus the inert discussions and projects steps).
• Incremental state (engine/incremental.rs): a watermark per repository in backup_state.json, advanced only for repositories whose every step succeeded, trusted only for the categories it was recorded with, set 15 minutes before the run started. Lists are always fetched in full and merged (backup/merge.rs); the watermark only decides which per-item requests (comments, events, commits, reviews) can be skipped. --full ignores it; an explicit --since replaces it and is never stored.
• Checkpoint: backup_checkpoint.json lists repositories finished in the current run; a later run resumes from it only if it is less than 6 hours old.
• Locking (lock.rs): an operating-system exclusive lock (fslock: flock / LockFileEx) on <owner>/json/.backup.lock; the CLI takes another on <output>/.github-backup.lock. The kernel releases them when the process dies, so there are no stale locks and no PID heuristics.
• Storage (storage.rs): every JSON file is written to a temporary name and renamed; release assets are streamed to .<file>.part and renamed after the size and digest checks.
• Git (git/): clones are made into a hidden staging directory and renamed into place; updates use fetch (mirror: --all --prune; bare: explicit refspecs; shallow: --depth; --lfs: mirror update plus git lfs fetch --all). Each git runs in its own process group with a stall timeout (600 s without output), is killed with its helpers on cancellation, and receives the token only through an environment variable and a host-scoped credential helper (credential.rs). -c safe.directory=<path> trusts exactly the repository being worked on.
• Cancellation (cancel.rs): a flag that kills running git processes and makes in-flight steps give way; the CLI sets it on SIGINT / SIGTERM.

github-backup-mirror

Post-processing: push the cloned repositories to another host.

• GiteaClient (REST v1) and GitLabClient (REST v4): check the destination, create it (private unless the source is known to be public), and verify that it is ours (description marker GitHub mirror of <owner>/<repo>) or empty.
• push.rs: git push --prune of refs/heads/* and refs/tags/* only (never --mirror), with the token in an environment variable and a credential helper scoped to the destination origin.
• runner / gitlab_runner: walk git/repos/*.git and report per-repository failures.

github-backup-s3

Post-processing: upload the JSON metadata to an S3-compatible store.

• signing: AWS Signature V4 built from sha2 + hmac (no AWS SDK).
• S3Client: PUT / HEAD / LIST / DELETE and multipart upload over hyper + rustls with retries and an idle timeout. It connects directly (no proxy support).
• sync_to_s3: concurrent sync of <output>/<owner>/json/; an object is skipped only when its stored SHA-256 digest (keyed HMAC for encrypted uploads) and size match; guarded --s3-delete-stale.
• encrypt: AES-256-GCM, wire format [12-byte nonce | ciphertext | 16-byte tag].

github-backup-tui

The Ratatui terminal interface (--tui). It drives the same BackupEngine through an event channel and does not embed its own backup logic. See the Interactive TUI guide.

github-backup (binary)

Flow of a normal run:

1. Parse the arguments and the optional config file (command line wins for single values; switches are OR-ed, lists unioned).
2. --tui, --list-scopes, --doctor/--check, --decrypt, --verify and --restore are handled and return before a backup.
3. Take the output lock, build the client, run BackupEngine::run, racing it against SIGINT / SIGTERM.
4. Post-processing, each step's failure being recorded, not fatal: manifest, diff, mirror push, S3 sync.
5. Write history, report and metrics, send the webhook, exit with 0 or 3.

A --dry-run stops after step 3's listing: it takes no lock and writes nothing.

Data Flow

GitHub API ──► GitHubClient ──► BackupEngine ──► Storage (FsStorage)  ──► <output>/<owner>/json
                                    │
                                    └──► GitRunner (git) ──────────────► <output>/<owner>/git
                                                                              │
                                  local backup ─────┬──► mirror push (git push to Gitea/GitLab)
                                                    └──► S3 sync (json/ only)

Concurrency Model

Repositories are processed as concurrent tasks bounded by a semaphore (--concurrency, default 4); owner-level data, gists and starred clones run first and sequentially. The API client, storage and git runner are Send + Sync. BackupStats uses atomic counters and a mutex-protected failure list.

Credential Security

See the Security page. In short: the token reaches git only through an environment variable and a credential helper scoped to the clone URL's origin; it is never in an argument list, a URL or a file, and log and error text is redacted.

Dependency Policy

deny.toml (enforced by cargo-deny in CI, in the release workflow and in a daily workflow) bans openssl, openssl-sys, native-tls and reqwest, allows only MIT, Apache-2.0 (also with the LLVM exception), ISC, BSD-3-Clause, Unicode-3.0, CC0-1.0 and Zlib, checks RustSec advisories and permits crates.io as the only source. TLS is rustls with the operating system's certificate store.

Unsafe Code

The workspace's own source contains no unsafe. github-backup-core has #![forbid(unsafe_code)]; the client, mirror, s3 and types crates have #![deny(unsafe_op_in_unsafe_fn)].

Testing Strategy

Restore Implementation

How --restore works internally. For using it, see the Restore Guide; this page does not repeat the procedures.

The code is crates/github-backup/src/restore.rs (the mode) and crates/github-backup-client/src/client/endpoints/write.rs (the three write calls: create_label, create_milestone, create_issue).

Control Flow

1. main parses the arguments and resolves the credential. With --restore it calls run::execute_restore instead of the backup: no engine, no post-processing, no lock, no contact with the source owner.
2. confirm_restore prints the warning banner and returns true if --restore-yes is set, GITHUB_BACKUP_RESTORE_YES=1 is set, or the user typed yes on a terminal. Without a terminal and without either escape hatch it prints both and the run ends with exit status 1. --dry-run skips the confirmation.
3. run_restore reads <output>/<owner>/json/repos/; if that directory does not exist the restore fails (exit 1: "no backup of '' found").
4. For every repository directory, restore_repo restores labels, then milestones, then issues. Errors are counted per resource kind and never abort the loop.
5. The totals are logged; exit status is 3 if any resource errored, else 0.

Per-Resource Rules

• Marker. Before restoring a repository's issues the tool pages through the target's issues (state=all) and collects every <!-- github-backup-restore:<owner>/<repo>#<number> --> found in a body. A backed-up issue whose marker is present is skipped. Each created issue's body is the original text, a rule, an italic "Restored by github-backup from ..." line (original author and creation time) and the marker.
• No restore of state. The create-issue request carries no state, assignees or milestone: closed issues are re-created open. Issue numbers are assigned by GitHub and differ from the originals.
• Missing target repository. Listing the target's issues fails, the repository's issues are counted as errored and skipped; its labels and milestones fail the same way, one request each.
• Order. Labels before issues so that label names exist. Label names are passed to the API as they are.
• Dry run. Counts what would be created, makes no request and does not look for markers.

The JSON is read through the typed models (Label, Milestone, Issue); the files are lossless copies of GitHub's responses, and these types ignore the properties restore does not need.

Why Pull Requests Are Not Restored

GitHub offers no API to create a pull request from historic data (authors, timestamps, review state, merged status). The data stays in pulls.json, pull_* and issue_* files for reference.

Tests

restore.rs has unit tests for the statistics, JSON loading, the body and marker format (including finding markers again), the error total, and a dry run that makes no API call. There is no automated test of the live marker lookup or of the exit statuses; those were exercised by hand against a local fake GitHub server during the documentation review, which is not part of the repository.

Contributing

Thank you for considering a contribution. This page explains how to get started and what to expect in review. (It is also the book page Development / Contributing.)

Development Setup

Prerequisites

• Rust 1.88 or newer (rust-version in Cargo.toml; install via rustup)
• git on the PATH (several tests run real git against local repositories)
• Optional: git-lfs, cargo-deny (cargo install cargo-deny --locked) and cargo-audit

Clone and Build

git clone https://github.com/tomtom215/github-backup-rust
cd github-backup-rust
cargo build --workspace --locked
cargo test --workspace --locked

Quality Gates

CI runs these; run them before opening a pull request:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --all-targets --locked
cargo +1.88 build --workspace --locked        # MSRV
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --locked
cargo deny check licenses bans advisories sources
cargo audit

The documentation book is built in CI with mdbook 0.4.40 and mdbook-linkcheck (mdbook build docs); a broken internal link fails the build.

Project Structure

See Architecture for the crate layout and the engine's failure model.

Coding Conventions

• No unsafe. The workspace's own source has none; github-backup-core forbids it (#![forbid(unsafe_code)]) and four crates deny unsafe_op_in_unsafe_fn. New unsafe needs explicit justification in review.
• No clippy warnings: CI treats them as errors. Fix them instead of adding #[allow(...)].
• Document public items (#![warn(missing_docs)] is set in the types, client, core, mirror and s3 crates; it is not set workspace-wide).
• Prefer small modules with one responsibility.
• Test new code: unit tests next to the code, or a *_tests.rs file; use MockClient, MemStorage and SpyGitRunner for engine tests.
• No OpenSSL, no reqwest: TLS is rustls, HTTP is hyper; deny.toml enforces it.
• Failures are recorded, not swallowed: a step that cannot be completed must end up in BackupStats (the engine's Steps do this) so that the run exits with status 3 and the report says so. Do not turn an error into a log line.
• Never put a secret in an argument list, a URL, a file or a log line.

Adding a New Backup Category

1. Add the field to BackupOptions (crates/github-backup-types/src/config/options.rs), and to BackupOptions::all() if --all should include it.
2. Add the key to ConfigFile (crates/github-backup-types/src/config/file.rs) and to crates/github-backup/src/config_template.toml.
3. Add the flag to crates/github-backup/src/cli/args.rs (and to the conflict list of --all).
4. Update merge_config_inner() and into_backup_options() in crates/github-backup/src/cli/args_impl.rs.
5. Add the API method to the BackupClient trait (crates/github-backup-client/src/api_client/mod.rs), implement it in crates/github-backup-client/src/api_client/impl_github.rs and crates/github-backup-client/src/client/endpoints/, and in the MockClient used by tests. Return Page<T> (the typed items with their original JSON) so that the file written is lossless.
6. Implement the backup function in a new module under crates/github-backup-core/src/backup/.
7. Run it as a step in crates/github-backup-core/src/engine/repo.rs (or engine/mod.rs for owner-level data).
8. Add unit tests with the mock client and MemStorage.
9. Document the flag in docs/src/backup-categories.md, docs/src/configuration/cli-reference.md, docs/src/configuration/config-file.md and docs/src/configuration/output-layout.md, and add it to the changelog.

Documentation

Every statement in the documentation must be true of the code: check it by running the binary or reading the code. Spelling is British English (organisation, behaviour, artefact), except in identifiers, JSON field names and GitHub's own terms. Do not hard-code version numbers in examples (use X.Y.Z); the single source is Cargo.toml.

Commits and Pull Requests

• Use the imperative mood in the subject line ("Add feature", not "Added feature") and keep it under 72 characters.
• Reference issues with Fixes #123 or Closes #123 in the body when applicable.
• Fork, branch from main, make focused commits with tests, run the quality gates and open a pull request against main; CI runs the same checks.

Reporting Issues

Open an issue at github.com/tomtom215/github-backup-rust/issues with the github-backup --version output (say whether it is a release or a build from main), the command you ran (redact tokens), the log output and your OS.

Security

Please do not open a public GitHub issue to report a security vulnerability. See SECURITY.md for the responsible disclosure process.

Code of Conduct

All contributors are expected to follow the project's Code of Conduct.

Security

The security design of github-backup, how credentials are handled, and recommendations for production. Each statement here was checked against the code; where something is a limitation it says so.

Credential Handling

Personal access tokens

• API requests carry Authorization: Bearer <token> (never a URL parameter) and go only to the API host you configured.
• Release-asset downloads send the token to GitHub's API host only: a redirect to another host, port or scheme is followed without the Authorization header, and a redirect from HTTPS to plain HTTP is refused.
• Git over HTTPS: the token is handed to the git child process in an environment variable and answered by an inline credential helper that is limited to the clone URL's scheme, host and port (-c credential.helper= first clears helpers you configured, so none can store the token). No file is written and the token is never part of an argument list, a remote URL or the repository configuration. A redirect, a submodule or a .lfsconfig that points at another host is never offered the token. The token is in the environment of the git process: the same user, and root, can read it from /proc/<pid>/environ while git runs.
• Mirror pushes use the same mechanism for the destination token.
• Logs and error messages are scrubbed of the token (and of well-known token prefixes) before they are printed, stored in --report, or sent to a webhook; the webhook payload carries no error message text for individual failures, only scope and step. -v and -vv do not print the token or request headers.
• Command line: a token passed with --token, an encryption key with --encrypt-key and S3 keys with --s3-access-key / --s3-secret-key are visible to every local user in the process list. Use the environment variables. The tool warns about --encrypt-key on the command line, not about the others.
• Debug output of the credential type is redacted.

Environment variables

The recommended way to supply secrets: GITHUB_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, MIRROR_TOKEN, BACKUP_ENCRYPT_KEY. They are visible to the same user and root through /proc; use a 0600 environment file (systemd EnvironmentFile=, docker --env-file) or your secret manager.

Config files

A token (or S3 or mirror secret) in a TOML config file works but is the least safe option. Mode 0600 is recommended; the tool logs a warning when the file is readable by group or others but does not refuse to use it.

Minimum Token Permissions

The scopes and fine-grained permissions per category are in Authentication. In short: a classic token with repo, gist, read:org and read:packages covers every category of --all; a fine-grained token is preferable when you do not need gists or packages. Fine-grained tokens can access an organisation's data when the organisation is the token's resource owner (and approves it), so organisation backups do not require a classic token.

Network Security

• TLS only for GitHub and the mirror hosts: the GitHub API client (which also serves --doctor), the OAuth device flow and the Gitea and GitLab API clients refuse plain http:// URLs; --api-url must be https://.
• Exceptions you control: an http:// S3 endpoint (for example a local MinIO) is allowed and logs a warning unless it is a loopback address; an http:// webhook URL is allowed and logs a warning. Both carry data in plaintext.
• TLS stack: rustls with certificates from the operating system's store (rustls-native-certs; SSL_CERT_FILE / SSL_CERT_DIR add trust). No OpenSSL; the dependency policy bans openssl, openssl-sys, native-tls and reqwest. There is no option to disable certificate verification. The crypto provider (aws-lc-rs) contains C code.
• Proxies: HTTP proxies only (via CONNECT); proxy credentials in the variable are sent as Proxy-Authorization and are visible in the environment. See Environment Variables.

Dependency Policy

deny.toml is enforced by cargo-deny in CI, in the release workflow and by a daily scheduled workflow:

Dependabot proposes weekly updates for Cargo, GitHub Actions and the Docker base images. The third-party GitHub Actions in the workflows are referenced by tag, not by commit SHA.

Release and Supply Chain

• Release binaries (five targets) carry a GitHub build-provenance attestation (see Installation). The attestation covers the binaries only; the container images, checksum files and SHA256SUMS.txt are not attested, and no SBOM is published.
• A release tag must point to a commit on main; the release workflow re-runs the tests, clippy, the MSRV build and cargo-deny before building.

S3 Credential Security

S3 credentials are accepted through --s3-access-key / --s3-secret-key / --s3-session-token or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN.

• Prefer the environment variables (command-line values show up in ps).
• Credentials and the session token are kept out of Debug output and logs; the secret key is wiped from memory when the client is dropped (best effort).
• Use the minimum IAM policy: s3:ListBucket on the bucket and s3:GetObject, s3:PutObject (plus s3:DeleteObject with --s3-delete-stale, and the multipart actions for large assets) on bucket/prefix/*; see S3.
• Enable bucket versioning where --s3-delete-stale is used.
• Encryption protects contents, not names, sizes, or swapping of objects; see At-Rest Encryption.

Integrity: What the Manifest Does and Does Not Do

--manifest records SHA-256 digests of the data files under json/ (not the history, state, checkpoint and lock files, which change on every run); --verify compares them. It detects accidental damage and casual edits. It does not cover the git clones, it is not signed (it sits next to the files it describes), and it has no digest of its own.

Output Directory Permissions

A backup can contain private repository code, webhook configuration, deploy key metadata, collaborator lists and security advisories. Restrict the directory:

mkdir -p /var/backup/github
chown backup-user:backup-group /var/backup/github
chmod 700 /var/backup/github

The tool does not set restrictive modes itself: new files follow the process umask. For multi-user systems consider disk encryption (LUKS or similar). The --encrypt-key feature protects only what is uploaded to S3.

Unsafe Code Policy

The workspace's own source contains no unsafe code. github-backup-core has #![forbid(unsafe_code)]; the client, mirror, S3 and types crates #![deny(unsafe_op_in_unsafe_fn)]; the binary and TUI crates carry neither attribute but contain no unsafe block. (Third-party dependencies such as hyper, rustls and aws-lc-rs do use unsafe.) There is no workspace lint table; the attributes are per crate.

Reporting Security Vulnerabilities

Please do not open a public GitHub issue for security vulnerabilities. Use GitHub's private security advisory feature:

1. Open github.com/tomtom215/github-backup-rust/security/advisories/new.
2. Describe the vulnerability and steps to reproduce.

See SECURITY.md for the disclosure policy, the in-scope and out-of-scope vulnerability classes and the expected response times.

Troubleshooting

The most common problems and how to resolve them. First look at the exit status and the summary at the end of the run (it lists every failure with its reason): 3 means the backup finished but something could not be backed up; 1 means it could not run. The same list is in the --report file. See Exit Codes and Monitoring.

Quick checks:

github-backup octocat --doctor        # git, output directory, network, token accepted?
github-backup octocat --dry-run --all # what would be backed up (writes nothing)

Authentication Errors

GitHub API error 401: Bad credentials

The token is missing, revoked, expired or mistyped. The run stops at once with exit status 1 and prints a hint.

• github-backup --doctor reports token ... rejected by GitHub (HTTP 401).
• Check the token the process really sees: GITHUB_TOKEN must be exported in the environment where github-backup runs (cron and systemd do not read your shell profile). An empty GITHUB_TOKEN= is treated as unset.
• Classic tokens need the scopes of the categories you enabled; see Authentication.

GitHub API error 403 / Resource not accessible

• The token lacks the scope or permission, or an organisation with SAML single sign-on has not authorised it (authorise the token in your GitHub token settings).
• For hooks, deploy keys, collaborators, advisories and branch protection a 403 or 404 on a repository is logged at INFO level (skipping hooks (no admin access)) and the file is simply not written; the run is not marked incomplete. If a file you expect is missing, look for skipping in the log.

The tool warns that the token "cannot read GET /user"

GitHub App installation tokens cannot call GET /user. For a user target the tool then lists public repositories and gists only. Use a personal access token to back up private data of your own account.

OAuth device flow ends with "OAuth device code expired"

You did not enter the code before it expired. Run the command again. "OAuth authorisation was denied" means you declined the request in the browser. --oauth-client-id is only valid with --device-auth.

Rate Limit Errors

rate limited; waiting Ns (attempt K)

A 429 or a 403 with rate-limit information (Retry-After, no remaining requests, or a "rate limit" message): the client waits as long as GitHub says (at least a minute, doubling, when GitHub gives no time) and retries, up to 6 times and about an hour of waiting per request. Nothing to do unless it takes too long.

rate limit exceeded; retry after Ns

The wait would exceed that budget (for example a primary limit that resets in more than an hour). The run stops with exit status 1 and keeps its checkpoint; run it again after the window resets. To reduce the pressure:

1. Lower concurrency: --concurrency 1.
2. Enable fewer categories, or split the work with --include-repos over several runs.
3. Leave the incremental state in place (backup_state.json): unchanged issues and pull requests then cost no per-item requests. (--since is not needed for this; it is an expert override.)

Network and TLS Errors

TLS error: ..., or system TLS roots fails in --doctor

No usable certificate store. On minimal systems install the CA package (apk add ca-certificates, apt-get install ca-certificates); the container image includes it. For a private CA set SSL_CERT_FILE (tool) and GIT_SSL_CAINFO (git).

HTTP transport error: client error (Connect)

The API cannot be reached. Behind a proxy set HTTPS_PROXY (an HTTP proxy; SOCKS is not supported):

export HTTPS_PROXY=http://proxy.example.com:3128
github-backup octocat --output /backup --all

When a proxy variable is present the log says HTTP proxy configured from the environment. git reads the same variables itself. S3 requests and the Gitea/GitLab API calls of --mirror-to ignore proxy settings and connect directly.

request timed out

Each request has a 120 s limit (headers, and the silence between two chunks of the body); transient timeouts and 5xx answers are retried (GET only, 3 times). There is no flag to change the limit. A git command that prints nothing for 10 minutes is stopped separately.

invalid API URL

--api-url must be an https:// URL with a host, for example https://github.example.com/api/v3.

Git Errors

git clone ... failed (exit 128): ... Repository not found

The token cannot see the repository (private repository without repo scope, SAML not authorised) or it was deleted or renamed. The run goes on with the other repositories and ends with exit status 3. --private is needed to include private repositories at all.

detected dubious ownership

git refuses a repository owned by a different user. The tool trusts the exact path it works on, so this is rare; if it happens run as the owning user (docker run --user, User= in the unit) or chown -R the output directory.

git ... made no progress for 600s and was stopped

A stalled connection. Re-run; check the network and proxy. A slow transfer that keeps printing progress is never stopped.

git lfs not found

--lfs needs git-lfs on the PATH (the container image has it).

git remote update / git fetch fails

The repository is logged as failed and the rest continue. Causes: network interruptions, a repository deleted or transferred since the last run, a directory owned by another user.

A branch I deleted on GitHub is still in the backup

By design: deleted branches and tags are kept. Pass --prune to remove them on the next run. See Clones Follow GitHub.

Storage Errors

cannot create output directory / Permission denied

--output must exist or be creatable, and be writable by the user running github-backup:

mkdir -p /var/backup/github
chown backup-user:backup-group /var/backup/github
chmod 750 /var/backup/github

In a container the volume must be writable by the container user (UID 1000 by default); see Docker.

another backup for '<owner>' is already running

Another github-backup holds the lock on that output directory. Wait for it. The lock is an operating-system lock that disappears when its process ends, so a crashed or killed run never leaves a stale lock; the .backup.lock and .github-backup.lock files that remain are harmless markers.

Disk full

The run stops (fatal error: stopping the run, exit status 1) and keeps its checkpoint. The tool does not check free space beforehand. Free space and re-run. A simple pre-check:

REQUIRED_GB=50
AVAIL_GB=$(df --output=avail -BG /var/backup/github | tail -1 | tr -d 'G ')
[ "$AVAIL_GB" -ge "$REQUIRED_GB" ] || { echo "Not enough disk space" >&2; exit 1; }

"0 repositories" in the summary

The listing came back empty or everything was filtered out. Check the OWNER spelling, --org for organisations, --forks / --private (forks and private repositories are excluded by default), --include-repos / --exclude-repos, and whether a user's token belongs to that user (private repositories need that).

S3 Sync Issues

A failed upload is a recorded failure (exit status 3) and the message names the S3 error code.

AccessDenied / 403

Check the keys, the endpoint, and the policy: it needs s3:ListBucket on the bucket and s3:GetObject and s3:PutObject on the objects (plus s3:DeleteObject for --s3-delete-stale). Without s3:ListBucket a HEAD of a missing object answers 403 and every file is uploaded on every run. See S3 permissions.

Objects are not updating

An object is skipped only when its stored SHA-256 digest and size match the local file. Changing the encryption key re-uploads everything. To force a full re-upload delete the objects (or use a new --s3-prefix).

Nothing happens in --dry-run

A dry run skips the S3 step entirely.

Mirroring Issues

401 / 403 on the destination

Check --mirror-token / MIRROR_TOKEN and that the token may create repositories (and push) at the destination.

A repository is refused as "foreign"

The destination already has a repository of that name that the tool did not create (its description is not GitHub mirror of <owner>/<repo>) and it is not empty. The tool never pushes into such a repository. Use another --mirror-owner, or empty it first.

422 from the Gitea API

The tool treats HTTP 422 on creation as "already exists" (a race between the existence check and the creation) and goes on to push. If the repository does not really exist, for example because the name is not valid at the destination, the push fails and is reported.

Enabling Debug Logging

github-backup octocat --output /backup --all -v 2>&1 | tee /tmp/debug.log

-v is debug, -vv is trace (connection-level events of the HTTP client; request and response headers are not logged, and no level prints the token). RUST_LOG overrides both.

Reporting Bugs

Please open an issue at github.com/tomtom215/github-backup-rust/issues and include:

1. The command you ran (redact tokens).
2. The relevant log output (-v) and, if you used --report, its failures.
3. Your operating system, and git --version.
4. The github-backup --version output (a build from main and the last release print the same number: say which one you use).

Changelog

All notable changes to github-backup are documented in this file.

The format follows Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Changes on main since v0.3.2 (2026-04-12). Nothing here is in a published release yet; github-backup --version prints 0.3.2 for both until the version is bumped.

Breaking changes and upgrade notes

• Exit status 3 is new. A run that finishes but could not back up everything now exits 3 (it used to exit 0 and report success). Scripts and unit files that treated every non-zero status as "could not run" need no change (a systemd unit simply fails); scripts that only checked for 1 should accept 3 as "incomplete". The other statuses are 0 ok, 1 fatal, 2 usage, 130/143 interrupted.
• Saved JSON is GitHub's complete response, with GitHub's key order, instead of a typed subset. Files are roughly four times larger and now contain properties that were dropped before (reactions, node_id, _links, review comment anchors, merge commit SHAs, ...). Pull requests also get issue_comments/<n>.json and issue_events/<n>.json.
• S3 object keys now include the owner: <prefix>/<owner>/json/<path> (they were <prefix>/<path>). The first run after upgrading uploads everything again under the new keys; the old objects are not removed (--s3-delete-stale only looks under the new <prefix>/<owner>/json/ root), so delete them yourself once you no longer need them. Objects are now skipped by SHA-256 digest plus size instead of size alone; an object without a stored digest is uploaded once more. With --encrypt-key the digest is keyed, so changing the key re-uploads everything.
• --mirror-to pushes branches and tags only (git push --prune with explicit refspecs, no --mirror), creates every repository private unless --mirror-public is given and the source is known to be public (a private source is never published), and refuses to push into an existing repository that it did not create (unless that repository is empty). Gitea repositories for an organisation owner are created through the organisation endpoint. Working-tree clones (--clone-type full) are not pushed.
• Git clones keep what GitHub deleted. Updates no longer prune: a branch or tag deleted on GitHub stays in the clone (force-pushed branches are still overwritten). Pass --prune (config key prune) for the old behaviour. --no-prune and the config key no_prune are still accepted and ignored; they cannot be combined with --prune. If a deleted branch foo is replaced by foo/bar, git cannot hold both: that update prunes once, logs a warning and continues. The TUI's "No Prune" toggle became "Prune deleted refs".
• --restore is restore-only: it no longer runs a backup first and works from the local backup. It asks for confirmation first, skips issues it restored before, and exits 3 when items could not be restored. Target repositories must exist.
• --keep-last and --max-age-days are deprecated and ignored (a warning is logged). They deleted directories by name pattern and could not work with the tool's own layout. Rotate copies with your backup tool.
• --since changed meaning. Issue and pull request lists are always fetched in full and merged into the stored files; --since DATE (now also YYYY-MM-DD) is an expert override that is never stored. v0.3.2 applied the previous run's timestamp automatically and could overwrite issues.json with only the changed issues. backup_state.json now holds a watermark per repository; an old file only costs one full fetch.
• --lfs is now a mirror clone plus git lfs fetch --all (it used to run git lfs clone, which created a non-bare checkout named <repo>.git whose refs never updated, and it overrode --clone-type). Delete and re-clone existing --lfs repositories to get a proper mirror.
• Report, metrics, webhook and history gained fields: see Added. success in the report now means "no failures" (it meant repos_errored == 0).
• Linux release binaries are static musl builds; the v0.3.2 x86_64 binary was glibc-linked and needed glibc 2.39 or newer.