How to Save on Logs and Not Go Crazy from Splunk Bills

Recently came across an interesting project called SigLens. The folks decided to take on something sacred — dethrone Splunk and ElasticSearch, promising tenfold resource savings in return. Sounds like typical marketing hype, if not for one "but": the project was recently archived, switched to Apache 2.0 license, and is now available in the open as a ready foundation for anyone who wants to build their own observability system without getting robbed.

By the way, SigLens developers claim their solution is 100 times more efficient than Splunk. I'm always skeptical about such numbers, but the team's technical background (serving 10,000+ engineers) makes you want to take a closer look at the details.

What's the Main Pain Point of Current Solutions

If you've worked with logs in large projects, you know this scenario. First you set up ElasticSearch, life is great, and then six months later you realize the cluster has grown to obscene sizes, requires constant admin attention, and devours memory like there's no tomorrow. Grafana Loki looks like salvation until complex queries come into play — performance sometimes resembles a turtle.

SigLens was conceived as an answer to these struggles. It's a Go binary that combines logs, metrics, and traces in one package. No external dependencies. You can run it on a regular laptop and, according to the authors, handle up to 8 TB of data per day.

What This Tool Can Do

The project is interesting primarily for its "all-in-one" architecture. Instead of jumping between different tools, you get a single entry point.

Support for Familiar Query Languages

This is probably its strongest side. If your team has been writing in Splunk SPL for years, there's no need to retrain. SigLens understands both SPL and regular SQL. This significantly lowers the entry barrier for migration.

Flexibility in Data Ingestion

The tool doesn't force you to rewrite your log delivery pipelines. It can accept data in OpenTelemetry, Elastic, Splunk HEC, and even Loki formats. Essentially, you can swap it in instead of your existing backend, and most of your agents (like Fluentbit or Vector) won't even notice the switch.

Performance in Real Numbers

The authors' blog has a case study where they processed 1 PB of data in a single day. For this, they needed just 32 EC2 instances. For comparison: Splunk or Elastic would require around 3,000 machines for the same task. The difference in infrastructure costs is colossal.

The Technical Side

SigLens is written in Go, which explains its lightweight nature. The main highlight here is the storage engine, optimized for the specifics of time series and unstructured logs. Unlike Elastic, which builds heavy inverted indexes on every field, SigLens uses more efficient approaches to compression and search.

Interestingly, the project offers built-in dashboards and an alerting system right out of the box. You don't necessarily need to bolt on Grafana, although such capability likely exists via API.

Who Could Benefit from It Now

Since the project has been moved to archive mode, it's a double-edged sword. On one hand, you shouldn't expect active development from the original team. On the other hand, the Apache 2.0 license gives you complete freedom.

I see several scenarios where SigLens could "take off":

1. Internal monitoring tools in startups, where infrastructure budget is limited but there's already a lot of data.
2. Local debugging. Thanks to it being a single binary, it's easy to spin up in a Docker container on a developer's machine for analyzing traces and logs in real time.
3. A foundation for creating your own proprietary data analysis solution.

Is It Worth Trying

If you're tired of paying thousands of dollars for Splunk or struggling with maintaining massive Elasticsearch clusters, SigLens is a great candidate to explore. Yes, the project is archived, but the Go code there is quite mature and functional.

For a quick start, they have ready-made Helm charts and Docker images. You can deploy the system in five minutes and see how it handles your log stream. Maybe this is exactly the case where "abandoned" open source can save a company a year's worth of cloud budget.

In my practice, it's often the case that these narrowly specialized engines perform better than universal all-in-one solutions. The main thing is to understand that you'll now have to provide support on your own or through the community that decides to fork the project.