ModelBest Runs a 35B Model on a Phone at 20 Tokens a Second
ModelBest has published the engineering behind its on device inference system, claiming a 35 billion parameter mixture of experts model running in real time on a phone at more than 20 tokens per second.
The write up covers Lookahead Expert Scheduling, or LES, the system behind MiniCPM-V 4.7 35B-A3B, and the company reports 20.51 tokens per second on an iPhone 18 Pro Max. It defines real time as decoding above 8 tokens per second, and lossless as keeping the original network structure, expert count and routing. Every figure is the company's own measurement.
And the constraint that shapes the whole design is memory. The model holds about 35 billion parameters and activates about 3 billion per token, and at 4 bit precision the weights alone need roughly 17.5GB, against 12GB of unified memory on the phone that the system and every app also draw on.
The answer is to keep expert weights in flash storage and load them as needed, which turns flash reads into the bottleneck. LES attacks that by feeding the current layer's router input into the next layer's router ahead of time, relying on how similar neighbouring layers are, so the reads start while the current layer is still computing. No separate predictor is trained, the company reports about 80 per cent recall of the experts it guesses, and the real router still makes the final choice.
Storage got the same treatment, with each expert's weights laid out as one continuous file region and a mixed precision split that keeps expert weights at Q3_K and the rest at Q4_K_M. ModelBest reports a five benchmark average of 78.12 against 78.28 at full precision and a weight file cut from 19.7GB to 14.44GB, so lossless here means the architecture and the routing survive, not that the output is bit identical.
But the last third of the gain comes from compute. The team adapted DSpark, a speculative decoding scheme from DeepSeek and Peking University, to llama.cpp and says the pull request has been merged upstream.
In the company's own ablation on a 12GB iPhone, decoding went from a 3.38 tokens per second baseline to 6.57 with LES, 11.34 once the scheduling and storage changes landed, 17.16 with the mixed precision weights and 20.51 with DSpark, which is 6.07 times the baseline. On an M4 iPad it measured 18.0 tokens per second with a larger expert cache and 7.8GB peak memory, and in its lowest memory setting it held peak memory near 3GB at 9.6 to 10.7 tokens per second.
The company has open sourced the LES code and the DSpark adaptation in a llama.cpp fork it names omni, and says the approach should carry to other mixture of experts models.
So the useful part of this paper is not the phone demo but the method, because prefetching the next layer's experts off the similarity between adjacent layers is a trick any model with too many parameters for its memory can try, and it costs nothing in accuracy by design.
ModelBest published the LES system behind MiniCPM-V 4.7 35B-A3B
It reports 20.51 tokens per second on an iPhone 18 Pro Max, 6.07 times its baseline
The model keeps expert weights in flash because 4 bit weights need about 17.5GB
LES predicts the next layer's experts with about 80 per cent recall before they are needed
The code and the DSpark adaptation are open source in the llama.cpp-omni repository
Source: Pandaily
