nestris-core Recognition engine & capture station¶
nestris-core reads Classic NES Tetris from a video signal. It finds the playfield in the picture, corrects perspective and lens distortion, and recognises score, lines, level, the next and current piece, the full 10×20 playfield and the piece statistics on every frame. It turns this into game events and statistics.
It is a high-performance Rust port of the earlier Python engine, wire-compatible with it and verified against it. It also contains nestris-station, the daemon that runs on every tournament station.
Ways to use it¶
-
nestris-station
A headless Debian daemon for tournament stations. It handles v4l2 capture, RFID login, cheat detection, validation, MQTT, recording upload, and runs under systemd with a watchdog. Shipped as
.debfor amd64 and arm64. -
nestrisCLIrun,replay,bench,verify,list-devices,screens. It writes JSONL, NDJSON or a WebSocket stream and records every game as NGF. -
Desktop GUIs
The egui GUI and the Qt 6 / QML GUI (styled after the NestrisChamps classic_1080 layout) each have three previews, a dashboard, a stats window and a settings dialog for every engine option.
-
Browser GUI
The engine compiled to WebAssembly (about 270 KB gzipped) runs in a Web Worker. It accepts video files, replays, a camera or a screen share, and downloads finished games as
.ngf.gz.
Recognition pipeline¶
flowchart LR
IN["<span style='color:#f8fafc'>Capture card · webcam ·<br/>video file · RTSP/SRT ·<br/>.ngf replay</span>"]:::hw --> FF["<span style='color:#111111'>ffmpeg</span>"]:::core
FF --> CAL["<span style='color:#111111'>Self-calibration<br/>RANSAC homography<br/>+ barrel undistortion</span>"]:::core
CAL --> CAN["<span style='color:#111111'>Canonical 256×240<br/>NES frame</span>"]:::core
CAN --> CLS["<span style='color:#111111'>Screen classifier<br/>(tile signatures)</span>"]:::core
CLS --> OCR["<span style='color:#111111'>Digits · pieces ·<br/>playfield · stats rail</span>"]:::core
OCR --> FUSE["<span style='color:#111111'>State fusion<br/>plausibility, events</span>"]:::core
FUSE --> INT["<span style='color:#111111'>Integrity<br/>cheat + validation</span>"]:::core
FUSE --> ST["<span style='color:#111111'>Statistics</span>"]:::core
INT & ST --> OUT["<span style='color:#ffffff'>JSON v4 · NGF · MQTT</span>"]:::ext
classDef core fill:#22d3ee,stroke:#0e7490,color:#111,stroke-width:2px
classDef ext fill:#64748b,stroke:#334155,color:#fff,stroke-width:2px
classDef hw fill:#1f2937,stroke:#94a3b8,color:#f8fafc,stroke-dasharray:4 3
- No manual calibration. A lock state machine (
UNLOCKED → ACQUIRING → LOCKED ⇄ DRIFT → LOST) finds the playfield and label anchors and re-solves in the background. Optional continuous tracking handles handheld or moving cameras. - Screen classifier. It matches the NES tile layout of every screen. On more than 60 000 hand-labelled station frames it agreed 100 %.
- Modded ROMs. Hexadecimal score displays are supported.
Verified against the reference engine¶
The Python engine is the reference. Equivalence is checked through four gates:
| Gate | What is compared | Result |
|---|---|---|
| 1 · CV golden tests | Computer-vision primitives vs OpenCV/numpy | within documented tolerances, many byte-exact |
| 2 · Recognition replay | Every field on Python-rectified frames | 100 % parity, every playfield cell included |
| 3 · State-machine replay | 15 fixtures × 1200 frames | byte-exact JSON, key order included |
4 · Full pipeline (nestris verify) |
End-to-end on all fixtures | ≥ 99.5 % per field required, most at 100.000 % |
Every new feature sits behind a configuration flag whose default reproduces the reference output byte for byte.
Performance¶
| Engine | p50 per frame | Throughput |
|---|---|---|
nestris-core (Rust, parallel) |
≈ 2.9 ms (p99 4.6 ms) | ≈ 350 fps |
| Python reference engine | ≈ 12.8 ms | ≈ 78 fps |
Runs are deterministic. RANSAC is seeded per frame, so two runs over the same input produce bit-identical output.
Statistics¶
Beyond the raw values, the engine computes tournament statistics. Full definitions are in the statistics reference.
- Base: pieces per second, score per minute, tetris rate, burn, drought / max drought, clear counts, pieces
- Extended: points breakdown by clear type, efficiency (EFF), pace to line 230, I-piece drought history, board shape (holes, tetris-ready, double well, clean slope), piece distribution
nestris-station in detail¶
flowchart LR
NES["<span style='color:#f8fafc'>🎮 NES</span>"]:::hw --> USB["<span style='color:#f8fafc'>USB capture<br/>v4l2</span>"]:::hw --> FF["<span style='color:#111111'>ffmpeg</span>"]:::core --> ENG["<span style='color:#111111'>engine</span>"]:::core --> SES["<span style='color:#111111'>session<br/>game start / end</span>"]:::core
RD["<span style='color:#111111'>RFID reader<br/>(serial JSON v2)</span>"]:::rfid --> PLY["<span style='color:#111111'>player</span>"]:::core --> SES
SES --> SPOOL["<span style='color:#111111'>disk spool</span>"]:::core --> MQTT["<span style='color:#ffffff'>MQTT → host</span>"]:::ext
SES --> REC["<span style='color:#111111'>.ngf.gz recording</span>"]:::core --> UP["<span style='color:#111111'>upload queue</span>"]:::core --> HTTP["<span style='color:#111111'>HTTP PUT → NestrisLTM</span>"]:::ltm
MQTT -. "cmd" .-> RD
classDef ltm fill:#f5b800,stroke:#7a5c00,color:#111,stroke-width:2px
classDef core fill:#22d3ee,stroke:#0e7490,color:#111,stroke-width:2px
classDef rfid fill:#f472b6,stroke:#be185d,color:#111,stroke-width:2px
classDef ext fill:#64748b,stroke:#334155,color:#fff,stroke-width:2px
classDef hw fill:#1f2937,stroke:#94a3b8,color:#f8fafc,stroke-dasharray:4 3
Install on Debian 12/13
sudo apt install ./nestris-station_<version>-1_amd64.deb # pulls in ffmpeg
sudoedit /etc/nestris-station/station.toml # capture.device, rfid.port, mqtt.host, station.id
sudo systemctl enable --now nestris-station
journalctl -u nestris-station -f # live log
The package installs the binary, a systemd unit and /etc/nestris-station/station.toml, and creates the nestris service user. Settings precedence: built-in defaults → config file → environment NESTRIS_STATION__<SECTION>__<KEY> → --set key=value. Unknown keys are errors.
| Section | Key settings |
|---|---|
station |
id (MQTT topic and game ID prefix), name |
capture |
device (device path or stream URL), resolution, stall_timeout_s |
rfid |
enabled, port, baud (115200) |
mqtt |
host, port, credentials, TLS, topic_prefix, live_max_hz (60) |
recording |
enabled, dir, keep_days (30), max_gb (20) |
host |
url (NestrisLTM), token_file (scope stations) |
integrity |
cheat detection and validation thresholds |
Self-healing
| Failure | Reaction |
|---|---|
| Process crash | systemd restarts it after 3 s |
| Main loop stuck | systemd watchdog (30 s) restarts it |
| Engine panic on a frame | engine rebuilt in-process; game flagged engine_restart |
| ffmpeg error / no frames | restart with exponential backoff |
| Capture stick unplugged | waiting_for_device, resumes on reconnect |
| Reader unplugged | reconnect loop, rfid: offline meanwhile |
| Broker / network down | results wait in the spool and are resent |
Release artifacts¶
| File | Platform | Contents |
|---|---|---|
nestris-station_<v>-1_amd64.deb / _arm64.deb |
Debian 12 / 13 | Station daemon |
nestris-v<v>-<os>-bundle.zip |
windows-x64, linux-x64, macos-arm64 |
CLI + GUIs bundle (Windows includes ffmpeg) |
nestris-cli-…, nestris-gui-…, nestris-qt-gui-v<v>-<os>.zip |
same | Individual tools; Qt as a folder, AppImage or .app |
nestris-web-v<v>.zip |
any static web host | Browser GUI |
SHA256SUMS.txt / .sig |
– | Checksums and Ed25519 signature |
nestris-core is published under the MIT License.