How to fit AI into smart glasses without going crazy from lag

Imagine you're building software for smart glasses or a smartwatch. You need the device to understand voice commands: "turn off the lights," "what's the weather tomorrow in St. Petersburg?" or "book a table." Usually, you'd call APIs of large language models like Gemini or GPT-4. But then problems start: a couple seconds of latency kills the entire user experience, and the bills for tokens grow faster than your user count.

The team at Cactus Compute decided to take a different approach. They took the knowledge from the massive Gemini 1.5 and "transferred" it into a tiny model called Needle, which is only 26 million parameters in size. For comparison: that's hundreds of times smaller than popular "small ones" like Llama-3-8B or even Qwen-0.5B.

What this tiny model can do

Needle isn't a chatbot that will discuss the meaning of life with you. It's a narrow-purpose tool for Function Calling. It takes a user's request in natural language and turns it into structured JSON that your code can understand.

The main feature here is speed. On their own infrastructure, the model delivers a blazing 6,000 tokens per second on prefill and 1,200 on generation. But even if you run it locally on a regular MacBook, it works practically instantly.

Here are a few things that make Needle an interesting project:

• The model was trained on 200 billion tokens. That's a serious amount for its size.
• It's optimized for edge computing — phones, wearables, and IoT devices.
• It comes with a convenient Playground where you can fine-tune the model for your specific tools in just a couple of clicks.

How it works inside

The architecture is called Simple Attention Network. If you look at the diagram that the authors left in the repository, you can see a rather elegant solution.

The model consists of a 12-layer encoder and an 8-layer decoder. Interestingly, the encoder has no traditional Feed-Forward networks (FFN) — it only has Self-Attention and Gated Residual connections. This saves a lot of memory and computational resources. The embedding weights and output linear layer weights are tied, which is also a standard trick for reducing model size without losing quality.

The authors honestly say: Needle is an experiment. It outperforms FunctionGemma-270m and Qwen-0.6B in function calling tasks, but falls short in regular conversations. Small models have little "memory" of the world, so it's easy to confuse them with a complex off-topic question.

Quick start and fine-tuning

Deploying the project is dead simple. After cloning the repository and running the setup script, command needle playground opens the web interface. There you can immediately test the model with your own examples.

If you need to embed Needle in a Python project, the code looks familiar:

from needle import SimpleAttentionNetwork, load_checkpoint, generate, get_tokenizer

# Загружаем веса и конфиг
params, config = load_checkpoint("checkpoints/needle.pkl")
model = SimpleAttentionNetwork(config)
tokenizer = get_tokenizer()

# Просим модель вызвать функцию
result = generate(
    model, params, tokenizer,
    query="Сколько градусов сейчас в Сочи?",
    tools='[{"name":"get_weather","description":"Узнать погоду в городе.","parameters":{"location":{"type":"string","description":"Название города."}}}]'
)
print(result)
# [{"name":"get_weather","arguments":{"location":"Сочи"}}]

The most valuable thing is the simplicity of fine-tuning. If you have specific APIs, you can "train" Needle on them. The authors recommend preparing at least 120 examples per tool to avoid overfitting. The repository even has a script for generating such data using the "older" Gemini.

Who should try it

I see Needle in projects where privacy and offline operation are critical. For example, in a smart home system that shouldn't send your conversations to the cloud, or in a mobile app for controlling a complex interface with your voice.

Of course, 26 million parameters impose limitations. You won't be able to ask it to write an essay or translate complex technical text. But if your task is to understand what the user wants to "turn it down" or "schedule a meeting for Thursday," Needle will handle it faster and cheaper than any other model.

The project is open source, the weights are on Hugging Face, and the MIT license allows you to use it in commercial products. If you've been looking for a way to add AI to a microcontroller or an old smartphone — this is a great candidate for testing.