How to Get Rid of the Container Zoo When Building AI Agents with SIE

Building autonomous agents on local or open-source neural networks is quickly becoming an infrastructure headache. If you're assembling a RAG system with multiple decision-making steps, you need to run several narrow-purpose systems simultaneously. You need a vector model for search, a separate reranker, a tool for parsing complex graphics in PDFs, a security classifier, and a generative LLM.

As a result, the dev environment quickly turns into a dump of docker-compose files. One container hogs memory for vLLM, another spins up Text Embeddings Inference, a third launches PyTorch just for paranoid entity extraction via GLiNER. Keeping all these instances running constantly is a surefire way to rack up massive GPU bills.

Engineers at Superlinked ran into the same problem and released SIE (Superlinked Inference Engine). This is an inference server that consolidates all agent task processing under a single umbrella.

How It Works Under the Hood

SIE takes a different approach. Instead of running five independent inference servers, you deploy one cluster that responds to standard OpenAI endpoints like /v1/embeddings or /v1/chat/completions.

The main feature is dynamic model loading on demand with an LRU (Least Recently Used) eviction algorithm. When an agent needs OCR to parse a user-uploaded scan, SIE loads the document recognition model into GPU memory. Once the parsing stage is complete and the system moves to dialogue, the rarely-used model frees up memory for the generative network.

The catalog comes with over a hundred pre-configured setups out of the box. Popular options include BGE-M3, ColBERTv2, SPLADE-v3, GLiNER, Docling, Qwen3, and prompt injection protection models like Granite Guardian.

Quick Start on Your Local Machine

For a first look, a standard Python package is enough. You can spin up a test instance on CPU or Apple Silicon in two commands:

pip install "sie-server[local]"
sie-server serve

If you're planning to run heavy workloads on NVIDIA GPUs, go with Docker from the start. The developers intentionally split the services into isolated containers due to conflicting system dependencies. OCR models require the latest transformers library, so they ship as a separate tag.

For vector search, launching the base container looks like this:

docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-default

You can verify it's working with a standard curl call:

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": "Привет, мир"}'

On first request, the server automatically downloads weights from Hugging Face and saves them to the local cache, so subsequent queries run without download delays.

Coding with the Python SDK

For working with the server, the authors wrote libraries for Python and TypeScript. The SDK handles calling specific tasks like classification or named entity extraction.

Here's an example of how to vectorize text, rerank results, and extract entities within a single client:

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

# Получаем эмбеддинг
embedding = client.encode("sentence-transformers/all-MiniLM-L6-v2", Item(text="Привет мир"))

# Считаем релевантность документов
scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text="Что такое машинное обучение?"),
    [Item(text="ML обучается на данных."), Item(text="Сегодня солнечная погода.")],
)

# Извлекаем сущности через GLiNER
entities = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text="Тим Кук руководит компанией Apple в Купертино."),
    labels=["person", "organization", "location"],
)

The syntax is straightforward. You don't need to write your own wrappers for HTTP endpoints or pull in third-party libraries for each minor model.

Production-Ready

Many similar open-source projects get stuck at the stage of a polished README for local use. In SIE's case, the authors immediately released the infrastructure tooling for deploying to Kubernetes.

The repository and related projects in the organization include:

• Helm chart sie-cluster for quick installation.
• Ready-made Terraform modules for AWS (EKS), Google Cloud (GKE), and Azure (AKS).
• KEDA configurations for autoscaling pods down to zero.
• Load balancer and Grafana dashboards for metrics collection.

The ability to scale pods down to zero comes in handy for internal corporate services. If employees aren't using the agent at night, cloud GPUs simply won't sit idle.

The Catch

The concept of a single inference server for all tasks sounds great, but there are no perfect solutions.

The main pitfall is cold start latency. When a model gets evicted from GPU memory due to LRU eviction, the next request has to wait for the weights to reload from disk. On relatively slow storage, this adds a couple of seconds of delay to the agent's response.

The second detail relates to Docker image separation. If your pipeline simultaneously needs OCR based on LightOnOCR and fast LLM inference on SGLang, you'll still need to spin up two different SIE containers since their environments differ.

Is It Worth Trying?

SIE is a great candidate for teams building pipelines on their own hardware or in a private cloud. If you're tired of propping up infrastructure held together with five different containers just for one RAG system, this project will save a ton of time.

If your architecture is simple and consists of only one conversational model without complex document processing and reranking stages, there's no point switching to SIE. Regular vLLM or Ollama will be perfectly sufficient in that scenario.