Skip to content

nestris-core Recognition engine & capture station

Latest release CI License Rust Beta

nestris-core reads Classic NES Tetris from a video signal. It finds the playfield in the picture, corrects perspective and lens distortion, and recognises score, lines, level, the next and current piece, the full 10×20 playfield and the piece statistics on every frame. It turns this into game events and statistics.

It is a high-performance Rust port of the earlier Python engine, wire-compatible with it and verified against it. It also contains nestris-station, the daemon that runs on every tournament station.

Repository Download latest

Ways to use it

  • nestris-station

    A headless Debian daemon for tournament stations. It handles v4l2 capture, RFID login, cheat detection, validation, MQTT, recording upload, and runs under systemd with a watchdog. Shipped as .deb for amd64 and arm64.

  • nestris CLI

    run, replay, bench, verify, list-devices, screens. It writes JSONL, NDJSON or a WebSocket stream and records every game as NGF.

  • Desktop GUIs

    The egui GUI and the Qt 6 / QML GUI (styled after the NestrisChamps classic_1080 layout) each have three previews, a dashboard, a stats window and a settings dialog for every engine option.

  • Browser GUI

    The engine compiled to WebAssembly (about 270 KB gzipped) runs in a Web Worker. It accepts video files, replays, a camera or a screen share, and downloads finished games as .ngf.gz.

Recognition pipeline

flowchart LR
    IN["<span style='color:#f8fafc'>Capture card · webcam ·<br/>video file · RTSP/SRT ·<br/>.ngf replay</span>"]:::hw --> FF["<span style='color:#111111'>ffmpeg</span>"]:::core
    FF --> CAL["<span style='color:#111111'>Self-calibration<br/>RANSAC homography<br/>+ barrel undistortion</span>"]:::core
    CAL --> CAN["<span style='color:#111111'>Canonical 256×240<br/>NES frame</span>"]:::core
    CAN --> CLS["<span style='color:#111111'>Screen classifier<br/>(tile signatures)</span>"]:::core
    CLS --> OCR["<span style='color:#111111'>Digits · pieces ·<br/>playfield · stats rail</span>"]:::core
    OCR --> FUSE["<span style='color:#111111'>State fusion<br/>plausibility, events</span>"]:::core
    FUSE --> INT["<span style='color:#111111'>Integrity<br/>cheat + validation</span>"]:::core
    FUSE --> ST["<span style='color:#111111'>Statistics</span>"]:::core
    INT & ST --> OUT["<span style='color:#ffffff'>JSON v4 · NGF · MQTT</span>"]:::ext

    classDef core fill:#22d3ee,stroke:#0e7490,color:#111,stroke-width:2px
    classDef ext fill:#64748b,stroke:#334155,color:#fff,stroke-width:2px
    classDef hw fill:#1f2937,stroke:#94a3b8,color:#f8fafc,stroke-dasharray:4 3
  • No manual calibration. A lock state machine (UNLOCKED → ACQUIRING → LOCKED ⇄ DRIFT → LOST) finds the playfield and label anchors and re-solves in the background. Optional continuous tracking handles handheld or moving cameras.
  • Screen classifier. It matches the NES tile layout of every screen. On more than 60 000 hand-labelled station frames it agreed 100 %.
  • Modded ROMs. Hexadecimal score displays are supported.

Verified against the reference engine

The Python engine is the reference. Equivalence is checked through four gates:

Gate What is compared Result
1 · CV golden tests Computer-vision primitives vs OpenCV/numpy within documented tolerances, many byte-exact
2 · Recognition replay Every field on Python-rectified frames 100 % parity, every playfield cell included
3 · State-machine replay 15 fixtures × 1200 frames byte-exact JSON, key order included
4 · Full pipeline (nestris verify) End-to-end on all fixtures ≥ 99.5 % per field required, most at 100.000 %

Every new feature sits behind a configuration flag whose default reproduces the reference output byte for byte.

Performance

Engine p50 per frame Throughput
nestris-core (Rust, parallel) ≈ 2.9 ms (p99 4.6 ms) ≈ 350 fps
Python reference engine ≈ 12.8 ms ≈ 78 fps

Runs are deterministic. RANSAC is seeded per frame, so two runs over the same input produce bit-identical output.

Statistics

Beyond the raw values, the engine computes tournament statistics. Full definitions are in the statistics reference.

  • Base: pieces per second, score per minute, tetris rate, burn, drought / max drought, clear counts, pieces
  • Extended: points breakdown by clear type, efficiency (EFF), pace to line 230, I-piece drought history, board shape (holes, tetris-ready, double well, clean slope), piece distribution

nestris-station in detail

flowchart LR
    NES["<span style='color:#f8fafc'>🎮 NES</span>"]:::hw --> USB["<span style='color:#f8fafc'>USB capture<br/>v4l2</span>"]:::hw --> FF["<span style='color:#111111'>ffmpeg</span>"]:::core --> ENG["<span style='color:#111111'>engine</span>"]:::core --> SES["<span style='color:#111111'>session<br/>game start / end</span>"]:::core
    RD["<span style='color:#111111'>RFID reader<br/>(serial JSON v2)</span>"]:::rfid --> PLY["<span style='color:#111111'>player</span>"]:::core --> SES
    SES --> SPOOL["<span style='color:#111111'>disk spool</span>"]:::core --> MQTT["<span style='color:#ffffff'>MQTT → host</span>"]:::ext
    SES --> REC["<span style='color:#111111'>.ngf.gz recording</span>"]:::core --> UP["<span style='color:#111111'>upload queue</span>"]:::core --> HTTP["<span style='color:#111111'>HTTP PUT → NestrisLTM</span>"]:::ltm
    MQTT -. "cmd" .-> RD

    classDef ltm fill:#f5b800,stroke:#7a5c00,color:#111,stroke-width:2px
    classDef core fill:#22d3ee,stroke:#0e7490,color:#111,stroke-width:2px
    classDef rfid fill:#f472b6,stroke:#be185d,color:#111,stroke-width:2px
    classDef ext fill:#64748b,stroke:#334155,color:#fff,stroke-width:2px
    classDef hw fill:#1f2937,stroke:#94a3b8,color:#f8fafc,stroke-dasharray:4 3

Install on Debian 12/13

sudo apt install ./nestris-station_<version>-1_amd64.deb   # pulls in ffmpeg
sudoedit /etc/nestris-station/station.toml                 # capture.device, rfid.port, mqtt.host, station.id
sudo systemctl enable --now nestris-station
journalctl -u nestris-station -f                           # live log

The package installs the binary, a systemd unit and /etc/nestris-station/station.toml, and creates the nestris service user. Settings precedence: built-in defaults → config file → environment NESTRIS_STATION__<SECTION>__<KEY> → --set key=value. Unknown keys are errors.

Section Key settings
station id (MQTT topic and game ID prefix), name
capture device (device path or stream URL), resolution, stall_timeout_s
rfid enabled, port, baud (115200)
mqtt host, port, credentials, TLS, topic_prefix, live_max_hz (60)
recording enabled, dir, keep_days (30), max_gb (20)
host url (NestrisLTM), token_file (scope stations)
integrity cheat detection and validation thresholds

Self-healing

Failure Reaction
Process crash systemd restarts it after 3 s
Main loop stuck systemd watchdog (30 s) restarts it
Engine panic on a frame engine rebuilt in-process; game flagged engine_restart
ffmpeg error / no frames restart with exponential backoff
Capture stick unplugged waiting_for_device, resumes on reconnect
Reader unplugged reconnect loop, rfid: offline meanwhile
Broker / network down results wait in the spool and are resent

Release artifacts

File Platform Contents
nestris-station_<v>-1_amd64.deb / _arm64.deb Debian 12 / 13 Station daemon
nestris-v<v>-<os>-bundle.zip windows-x64, linux-x64, macos-arm64 CLI + GUIs bundle (Windows includes ffmpeg)
nestris-cli-…, nestris-gui-…, nestris-qt-gui-v<v>-<os>.zip same Individual tools; Qt as a folder, AppImage or .app
nestris-web-v<v>.zip any static web host Browser GUI
SHA256SUMS.txt / .sig – Checksums and Ed25519 signature

nestris-core is published under the MIT License.