Create a profile

A profile is pure configuration — no audio, no code. It says exactly how a benchmark runs and is scored: which runtime, which model, which language, which metrics. Prefer the in-browser builder to skip hand-writing YAML — this page is the reference for what each field means.

Fields

Validated against schemas/benchmark-profile.schema.json in the runner repo.

Field	Required	Notes
id	Yes	Lowercase, hyphenated, unique, permanent — e.g. whisper-medium-nl-batch.
version	Yes	Semver (1.0.0). Any scoring/run-logic change needs a new version — profiles are never edited in place.
title	No	Free-text display name.
benchmark_type	Yes	batch, streaming, or conversation.
language	No	BCP-47 (nl, nl-NL, en-US, ...). Any language is supported — none is hardcoded.
runtime.name	Yes	The adapter id — faster-whisper, whisper.cpp, or vosk today.
runtime.min_version	No	Minimum adapter version this profile assumes.
model.name	Yes	e.g. whisper-medium.
model.quantization / beam_size / temperature / vad	No	Model-level knobs; only set what the runtime actually uses.
configuration.threads / chunk_ms / streaming	No	Runner-level execution settings.
normalization.ruleset_id	No	Per-language scoring ruleset, e.g. goesb-en-v1, goesb-nl-v1. New language usually means a new ruleset too.
scoring.primary_metric	Yes	The metric leaderboards rank by, e.g. wer.
scoring.tie_breakers	No	Ordered list of metrics used to break ties.
metrics	Yes	Array of metric ids this profile reports (min. 1).
changelog	No	Array of {version, notes} — required once you bump version past 1.0.0.

Worked example

A real official profile — Whisper Medium, Dutch, batch:

id: whisper-medium-nl-batch
version: 1.0.1
title: Whisper Medium Dutch (Batch)
benchmark_type: batch
language: nl-NL
runtime:
  name: faster-whisper
  min_version: "1.0.0"
model:
  name: whisper-medium
  quantization: int8
  beam_size: 5
  temperature: 0.0
  vad: true
configuration:
  threads: 4
normalization:
  lowercase: true
  remove_punctuation: true
  expand_numbers: true
  ruleset_id: goesb-nl-v1
scoring:
  primary_metric: wer
  tie_breakers: [real_time_factor, energy_wh]
metrics: [wer, cer, real_time_factor, cpu_pct, ram_mb, energy_wh, temperature_c]
changelog:
  - version: 1.0.0
    notes: Initial profile.
  - version: 1.0.1
    notes: Added temperature_c (M2) now that the hwmon probe exists.

Submit it

No local git setup needed for a single file — the GitHub web editor works fine:

Fork taktx-io/GOESB on GitHub.
In your fork, create profiles/<your-id>/profile.yaml (press . on the repo page to open the in-browser editor, or use Add file → Create new file).
Paste your YAML, matching the fields above.
Commit directly to a new branch on your fork, then open a pull request.

Prefer the command line? Clone the repo (see the how-to guide), add your file, then:

goesb validate profiles/<your-id>/profile.yaml
python scripts/validate_assets.py

git checkout -b add-<your-id>-profile
git add profiles/<your-id>/profile.yaml
git commit -m "Add <your-id> profile"
git push -u origin add-<your-id>-profile
# then open a PR from your fork on GitHub

A reviewer checks it against the same criteria as any code contribution — security, neutrality, correctness, reproducibility — see the Contribute page.