PrismML shrinks 27B AI model enough to run on an iPhone

PrismML has released Bonsai 27B, a compressed artificial intelligence model that the company says can run on an iPhone 17 Pro Max. For investors tracking the AI buildout, the notable part is not another chatbot demo, but a potential shift in where AI workloads can run: on consumer devices instead of only in cloud data centers.

The model has 27 billion parameters, the adjustable values that help an AI system process prompts and generate answers. A model of that size would typically need about 54 GB of memory at half precision, according to the figures cited by PrismML, putting it out of reach for most phones and many personal computers.

PrismML says Bonsai 27B’s smallest version is 3.9 GB and runs at 11 tokens per second on an iPhone 17 Pro Max. Tokens are the chunks of text or data that an AI model reads and produces. A separate ternary version is 5.9 GB and reaches about 26 tokens per second on an M5 Pro laptop, according to the company.

How the compression works

The main technical move is aggressive compression of model weights, the internal numbers that store what the model learned during training. PrismML says the method is based on Caltech intellectual property and cuts each weight from 16-bit floating-point precision to a much smaller representation.

In the binary version, each weight is reduced to a sign, either positive or negative. In the ternary version, each weight can be negative, zero, or positive. Groups of 128 weights share a 16-bit scaling factor, which PrismML says brings the binary model to 1.125 bits per weight, about 14 times smaller than the full-precision original. The ternary model lands at 1.71 bits per weight.

That sounds abstract, but the basic idea is straightforward. A standard 16-bit AI model can use about 65,000 possible settings for a value. A ternary model uses three. The bet is that the model can keep much of its useful behavior even after those values are heavily simplified.

PrismML says Bonsai differs from many low-bit AI models because the compression applies across the full model, including embeddings, attention layers and the language model head. Low-bit models are compressed versions of larger models that use fewer bits to store weights, often trading some accuracy for smaller size and lower memory use. PrismML says Bonsai does not keep certain layers at higher precision to protect quality, a common approach that can make models larger.

Benchmarks and broader implications

PrismML says the ternary Bonsai 27B retained 94.6% of the full-precision model’s benchmark performance. Across 15 benchmarks run in thinking mode on NVIDIA H100 GPUs, covering knowledge, math, coding and tool use, the ternary model averaged 80.49, while the 1-bit version scored 76.11, according to the company.

The company previously released Bonsai 8B in March, a 1.15 GB model designed to show that its 1-bit approach could work at 8 billion parameters. Moving to 27 billion parameters puts the technology into a larger class of models associated with stronger reasoning, tool use and multi-step behavior, according to PrismML’s framing.

Both Bonsai 27B versions are available free under the Apache 2.0 license, PrismML said. CNBC reported that Apple is in early talks with PrismML about the compression technology. PrismML is also targeting a compressed Gemma model next, according to the same reporting.

On-device AI can reduce reliance on remote servers, which may affect costs, privacy design and app performance. PrismML’s release does not prove that large models will leave the cloud, but it adds another data point to a key AI question: how much intelligence can fit into the hardware people already carry.

This story draws on original reporting from Decrypt.