How to Run a 27 Billion Parameter Neural Network on a Regular iPhone or Home PC

Recently I caught myself thinking: we've gotten used to the idea that running serious LLMs (Large Language Models) requires server racks with A100s or at least top-tier RTX 4090s. But what if I told you that a 27B-level model can now be "squeezed" into the RAM of a regular smartphone or laptop without discrete graphics? The Bonsai-demo project from the PrismML team does exactly this, using the magic of 1-bit and ternary quantization.

What's the Core of the Project

The main problem with large models is their "greediness." A standard 27B model in 16-bit format weighs about 50 GB. Even popular 4-bit compression (Q4_K_M) requires 16-17 GB of video memory just for the weights. Bonsai-demo offers moving to an extreme level: 1-bit and ternary (1.58-2 bits) models.

As a result, Bonsai-27B in 1-bit configuration takes only 3.5 GB. This is comparable to the size of an average mobile game. The developers claim that the model retains the ability to reason, see images, and even call tools.

What This "Tree" Can Do

I dug into the repository and highlighted a few things that really stand out.

Vision and Document Processing

The 27B series models here are not just text-based. These are multimodal systems. You can feed them screenshots, photos, or PDF files. Inside, a special projector is used that converts visual data into tokens the model can understand. For a local system running on CPU, this looks like magic.

Agent-like Behavior and MCP

The demo includes a full-fledged client for the MCP (Model Context Protocol). If you missed it: this is a new standard from Anthropic that allows the model to connect to external data. Bonsai-demo already has out-of-the-box tools configured for working with DeepWiki and Hugging Face. So the model doesn't just hallucinate—it goes to the internet for facts.

"Thinking" Mode

The 27B models have a chain-of-thought reasoning feature. In the interface, you can set a thinking budget: from a quick answer to deep analysis of 8,000+ tokens. If your hardware is weak, it's better to set it to Low, otherwise you'll have to wait a long time while the model "spins" its thoughts in the background.

Extreme Context Savings

Usually, a context of 100,000 tokens consumes gigabytes of memory. Here, the authors added experimental support for 4-bit KV-cache. This reduces memory consumption for long conversations by 3.5 times. In numbers: 100k tokens take about 1.8 GB instead of the usual 6.3 GB.

Technical Details

The project relies on a fork of llama.cpp and the MLX framework for Apple Silicon. Interestingly, 1-bit quantization support (Q1_0) has already started flowing into the main llama.cpp upstream. With ternary weights (Q2_0), the situation is slightly more complex: they are currently in the process of migration, so the project ships its own precompiled binaries for different platforms.

If you have a Mac with an M-chip, the build script will pull MLX and compile all components right for your architecture. For Windows and Linux, scripts with auto-detection of CUDA and Vulkan are provided.

How to Try It

The developers made the process as "seamless" as possible. No need to manually download weights from Hugging Face or configure the environment.

For macOS or Linux, just three lines are enough:

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
./setup.sh

The script will automatically install the package manager uv, create a virtual environment, download the required model (Ternary-Bonsai-27B by default), and prepare the binaries.

After that, you can start a local server:

./scripts/start_llama_server.sh

And open localhost:8080 in your browser. There will be a familiar chat where you can test both vision and generation speed.

Is It Worth Trying

Bonsai-demo is not about "killing GPT-4," but about accessibility. If you need to run an intelligent assistant on an office laptop without a graphics card or embed an LLM into a mobile application, this project provides a ready-made foundation.

Of course, 1-bit models are dumber than their full-sized counterparts. They may make more mistakes in complex logical tasks. But for basic automation, text classification, or simple chatbots with a context of 256k tokens—this is one of the most efficient solutions available right now.

The main advantage of the repository is its "packaged" nature. You don't need to be a quantization specialist to run a ternary model. All the dirty hacks with compilation and parameter tuning are already hidden inside the bash scripts.