GOESB

Run a benchmark, step by step

The runner is a local command-line tool — your audio and models never leave your machine. Producing a result never needs network access; only the last step, submitting it, does.

Prefer not to memorize flags? After installing (step 1), just type goesb with no arguments — an arrow-key menu walks you through picking a profile, a pack, and running or submitting a benchmark, covering steps 2–4 below.

1. Install the runner

Three ways to get it — pick whichever fits:

PyPI (recommended) — needs Python 3.11+

Check your version first — python3 --version (Windows: py --version). Then install via pipx — gives it its own isolated environment automatically, so it works cleanly even on Debian/Ubuntu (which blocks a bare pip install outside a venv by default):

# if you don't have pipx yet:
sudo apt install pipx && pipx ensurepath        # Debian/Ubuntu
brew install pipx                               # macOS
py -m pip install --user pipx && py -m pipx ensurepath   # Windows

pipx install "goesb-runner[faster-whisper]"
# or: pipx install "goesb-runner[vosk]"
# or: pipx install "goesb-runner[whisper-cpp]"

Prefer a plain venv instead? python3 -m venv .venv && source .venv/bin/activate && pip install "goesb-runner[faster-whisper]" (Windows: .venv\Scripts\Activate.ps1) works the same way — inside an activated venv, plain pip/python already point at the right one. Outside a venv on a system where python/pip still resolve to Python 2, use python3/pip3 instead. Didn't pick the right engine, or want to try another? No need to reinstall — picking a profile that needs an engine you don't have yet prompts to install it on the spot.

Standalone binary — no Python needed at all

One binary per engine (smaller downloads, only the dependencies that engine needs) — download from the latest release:

goesb-faster-whisper-macos-arm64      # or -linux-x64, -linux-arm64, -windows-x64.exe
goesb-vosk-macos-arm64
goesb-whisper-cpp-macos-arm64

macOS/Linux: chmod +x the file after downloading. Every command below is the same, just run ./goesb-faster-whisper-macos-arm64 (or whichever you downloaded) instead of goesb.

From source — for contributing, or authoring a new pack

git clone https://github.com/taktx-io/GOESB.git
cd GOESB
python3 -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e "./runner[dev,faster-whisper]"

whisper-cpp builds a native extension and needs a C++ toolchain (Xcode Command Line Tools + CMake on macOS, build-essential cmake on Debian/Ubuntu, Visual Studio Build Tools + CMake on Windows) — the other two engines install prebuilt wheels and need nothing extra.

Upgrading

Check what you have with goesb version, then upgrade with whichever method you installed with:

pipx upgrade goesb-runner              # pipx
pip install --upgrade "goesb-runner[faster-whisper]"   # plain venv (swap in your engine)
git pull                               # from source — editable install picks it up automatically

Standalone binary: there's no upgrade command — download the new one from the latest release and replace the old file. If pipx upgrade reports no change but goesb version still looks stale, force it: pipx install --force goesb-runner.

2. Pick a profile and pack

List what's available instead of guessing an id:

goesb list-profiles
goesb list-packs

GOESB never hosts audio (privacy-first), so where the audio for a pack comes from depends on the pack:

  • Official open packs (built from FLEURS or LibriSpeech, both plain ungated downloads) — nothing to do here. goesb run in step 3 fetches the audio automatically, no clone or separate script needed.
  • Already have your own audio? Bring your own pack.yaml (the in-browser builder makes one from your files) and point --audio-dir at it in step 3.
  • Anything else (a private/custom pack, or a corpus with a consent step like Common Voice that can't be scripted) — goesb run can't auto-fetch it and prints that pack's own manual fetch instructions instead.

3. Validate and run a benchmark

validate checks a local file you have in hand — useful while authoring a new profile/pack, not needed for an existing official one. run is the one that fetches: give it an official profile/pack id and it pulls whatever isn't already local — profile, pack metadata, and (for a known-source open pack) the audio itself:

# --model-override tiny keeps this fast for a first run
goesb run whisper-medium-en-batch librispeech-en-batch --model-override tiny

This writes a signed, hashed result document to runs/results/. Fetched profiles and packs are cached under ~/.goesb/cache; auto-fetched audio lands right next to the pack (packs/<pack_id>/audio locally, or alongside the cached pack when it was fetched too) — either way, a second run of the same one is fully offline. Add --offline to refuse any network access and fail clearly if something isn't already local. --api-url overrides where it fetches from (defaults to https://www.goesb.com/api). Bringing your own audio? Add --audio-dir pointing at it.

4. Submit your result

A fresh, single-use signing token is requested automatically — your private key never leaves your machine.

goesb submit "$(ls -t runs/results/*.json | head -1)"

$(ls -t runs/results/*.json | head -1) picks the most recently produced result automatically (macOS/Linux) — or substitute the exact filename the previous step printed. submit checks its target is actually compatible with this runner's version before sending anything, and tells you plainly to upgrade if not — see "Upgrading" in step 1 above.