How to Build a Zoo of AI Agent Tools into a Single Stack

Anyone who has tried to deploy autonomous agents or complex LLM pipelines to production quickly hits the same pain point. First, you bolt on Langfuse or Phoenix for tracing. Then you realize that logs without automated evaluation are useless, so you drag in a separate library for evals. Next, you discover that the agent is happily catching jailbreaks and leaking tokens, so you layer Guardrails or Lakera on top. And you still need a fast gateway for routing between providers and a simulator for running through dialogues before deployment.

As a result, instead of working on the product, teams spend weeks duct-taping five different services together. The future-agi project tries to solve this chaos with a single box. The team built an open-source platform under the Apache 2.0 license that bundles tracing, evals, simulations, a gateway, and prompt optimization.

What's Inside the Box

The core idea behind the platform is simple: combine production data collection and agent improvement into a single closed loop. If an agent made a mistake on a real user, that trace should immediately become a test case for the next prompt iteration.

All functionality is split into six main areas.

Scenario Simulation and Synthetic Tests

Testing agents manually is slow and inefficient. The built-in simulation module can generate thousands of multi-step dialogues with different roles, tricky questions, and hacking attempts.

What's interesting is that the module supports not only text but also voice agents through integrations with LiveKit, Retell, VAPI, and Pipecat. You can check latency and voice activity detector (VAD) performance before a live customer hears the bot's incoherent mumbling.

Real-time Quality Assessment and Safety

The library includes over 50 built-in metrics. There are standard checks for hallucinations and context adherence, tool use correctness evaluation, tone analysis, and personal data leakage detection. Evaluation works through a combination of heuristics, lightweight ML models, and the LLM-as-a-judge approach.

For real-time protection, there are 18 built-in scanners against injections and jailbreaks, plus adapters for external solutions like Presidio or Llama Guard. Scanners can be invoked directly within the proxy gateway or called via a separate SDK in your application code.

Command Center Gateway and Tracing

Request routing is handled by a gateway written in Go. The authors claim weighted routing latency of around 9.9 nanoseconds and throughput of about 29,000 requests per second on a t3.xlarge instance. With security checks enabled, P99 latency stays around 21 milliseconds.

The gateway is OpenAI API-compatible, supports over a hundred providers (from OpenAI and Anthropic to local Ollama and vLLM), can do semantic caching, and manages virtual keys.

For tracing, the OpenTelemetry standard is used. The platform collects span graphs, latency, and token costs from 50 frameworks like LangChain, CrewAI, DSPy, or LlamaIndex without complex manual configuration.

Prompt Optimization on Real Data

The most interesting part is algorithmic prompt tuning. Instead of manually tweaking wording in code, six optimization algorithms are available, including GEPA, PromptWizard, ProTeGi, and Bayesian search. Real production error traces are fed into the optimizer, which automatically generates and tests new system prompt variants.

Quick Start

The project can be deployed locally via Docker Compose or run in the cloud. For local deployment, just clone the repository and run the installation script:

git clone https://github.com/future-agi/future-agi.git
cd future-agi
./bin/install

After completion, the web interface is available at http://localhost:3000.

Integration into existing Python code takes literally three lines:

from fi_instrumentation import register
from traceai_openai import OpenAIInstrumentor
import openai

register(project_name="support-agent")
OpenAIInstrumentor().instrument()

client = openai.OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Как оформить возврат товара?"}],
)

In a TypeScript or Node.js codebase, everything looks similar:

import { register } from "@traceai/fi-core";
import { OpenAIInstrumentation } from "@traceai/openai";
import OpenAI from "openai";

register({ projectName: "support-agent" });
new OpenAIInstrumentation().instrument();

const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Как оформить возврат товара?" }],
});

After the call, data automatically goes to the tracer where you can see the call tree, passed parameters, and measure generation time.

Architecture and Tech Stack

Under the hood, the platform uses a fairly practical stack with no exotic choices:

• Backend written in Python 3.11 using Django 5.1 and Django Channels for WebSockets.
• High-load gateway written in Go 1.23.
• Frontend built with React 18 and Vite bundler.
• PostgreSQL is used for metadata and configuration storage, ClickHouse for span analytics and time series, Redis for state and cache. Background tasks run via RabbitMQ and Temporal.

All ecosystem components are split into separate packages. If you don't need the entire interface, you can install just the metrics module pip install ai-evaluation or the prompt optimizer pip install agent-opt.

Who This Project Is For Right Now

If you're building a simple FAQ chatbot that answers questions with one button, Future AGI will seem like an overkill combine. A couple of basic console logs will do just fine there.

But if you're building:

• Complex RAG pipelines where you need to validate citations and fact-check answers.
• Voice assistants based on LiveKit or Retell where every millisecond of latency is critical.
• Agents calling dozens of external APIs and MCP tools where catching failures in reasoning chains is critical.
• Systems that can't be handed off to external SaaS services due to data isolation requirements.

In these scenarios, being able to spin up the entire monitoring, simulation, and gateway stack with one command docker compose up saves a lot of engineering hours.

The project is currently in active development, as the banner in the README honestly warns. You may encounter some rough edges in the interface here and there, but the open-source code and architecture based on OTel and ClickHouse make it a great candidate for testing in your own infrastructure.