How to Stop Writing Adapters for Neural Networks and Take Control of Token Costs

Recently, I was rewriting the Claude integration to an updated client and caught myself thinking. One day clients ask to connect GPT-4o, another day they demand Anthropic, and a week later the finance department asks where a several-hundred-dollar bill for tests came from. Each time I have to add error handling logic, manage keys, and manually calculate token expenses.

This routine is solved by LLM Gateway from The Open Co team. The project serves as a unified API gateway that accepts calls in the standard OpenAI format and routes them to the appropriate providers.

One Request for Any Model

The core concept is straightforward. Rather than integrating multiple SDKs, you send a single HTTP request to a local or cloud gateway. The controller automatically identifies the target provider, transforms the format, and returns the response.

Currently, the main providers are supported:

• OpenAI
• Anthropic
• Google Vertex AI
• Other services with compatible APIs

Here's what a standard gateway request looks like:

curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -d '{
  "model": "gpt-4o",
  "messages": [
    {"role": "user", "content": "Hello, how are you?"}
  ]
}'

If you need to switch to Claude 3.5 Sonnet, the JSON structure in your application stays the same. Only the model name in the request body changes.

Cost Tracking and Latency Metrics

When multiple services or developers work with neural networks, controlling limits becomes difficult. Sometimes someone runs a script with an incorrect prompt in an infinite loop and burns through a month's budget in an hour.

The gateway takes care of tracking. Each transaction is saved to the database, and the system automatically calculates:

• Number of input and output tokens
• Total cost of each call
• Model response time
• Overall statistics by keys and projects

Through the web panel, you can view ready-made graphs and immediately see which specific model is consuming the bulk of the budget.

Project Structure and Running in Docker

The authors built a monorepo in TypeScript. Under the hood, proven technologies are used:

• Hono handles API request proxying
• Next.js manages the web interface and playground
• Drizzle ORM works with PostgreSQL and Redis databases
• TypeScript ensures end-to-end typing of components

You can deploy your own service in a couple of minutes via Docker. The authors assembled a ready-made image combining the main components.

docker volume create llmgateway_postgres
docker volume create llmgateway_redis

docker run -d \
  --name llmgateway \
  --restart unless-stopped \
  -p 3002:3002 \
  -p 3003:3003 \
  -p 3005:3005 \
  -p 3006:3006 \
  -p 4001:4001 \
  -p 4002:4002 \
  -v llmgateway_postgres:/var/lib/postgresql/data \
  -v llmgateway_redis:/var/lib/redis \
  -e AUTH_SECRET="$(openssl rand -base64 32 | tr -d '\n')" \
  -e GATEWAY_API_KEY_HASH_SECRET="$(openssl rand -base64 32 | tr -d '\n')" \
  ghcr.io/theopenco/llmgateway-unified:latest

A small detail from the documentation: do not mount a folder from the host machine directly into /var/lib/postgresql/data. Due to the specifics of PostgreSQL permission initialization in the container, the process may crash. The named volumes in the command above eliminate this issue.

If you want to try the system first without deployment, the developers have a cloud version at llmgateway.io.

Free Version Limitations

The repository uses dual licensing. The main code is distributed under AGPLv3, however some folders in the source code belong to the Enterprise version.

In the free open-source version, call history is stored for 30 days. If you need unlimited log retention, advanced user billing, or team separation within your organization, you'll need to purchase a commercial license.

Who Will Benefit from This Tool

If your application makes three requests per day to a single model, there's no point in setting up a separate proxy. You'll just add an extra failure point and negligible network latency.

The gateway will prove itself in the following situations:

• The project uses models from different providers
• Transparent token cost tracking across different services is required
• Proxy deployment in your own environment is needed
• A quick fallback switch to a backup model in case of failures is planned

You can try the project on GitHub. The README there is quite minimal, but the project is understandable even without lengthy instructions.