671Bparameters
Running on one consumer GPU: a single RTX 5090.
- Problem
- Run language models far larger than available GPU memory on consumer workstation hardware.
- Engineering
- A from-scratch CUDA C++ inference runtime, built rather than assembled from existing serving stacks.
- Result
- A 671B-parameter model runs on a single RTX 5090, and a 397B-parameter model runs across two consumer GPUs at 23.3 tokens per second with its active expert set fully resident.
Hire us for
Data and applied AI. Models and inference running on hardware you own, inside your building, with nothing sent to a third party.
