How to Process Billions of Events per Second Without Killing Latency

Imagine this: you need to check bank transactions for fraud in real time during the payment process. You have only a few milliseconds to make a decision. Apache Kafka is streaming events non-stop, but for each one you need to pull up the customer's history from the database. If you query a traditional PostgreSQL or MySQL for every message, the system will collapse under the load instantly.

This is a common dead end engineers hit when designing high-load systems. A regular cache like Redis helps speed up individual key reads, but when it comes to building complex business logic and analytics on top of a data stream, its capabilities start to fall short. You're forced to tie together caches, message brokers, and third-party processing engines. At this point, it makes sense to take a look at Hazelcast.

Hazelcast combines distributed in-memory storage and a stream processing engine in a single system. Instead of assembling a construct from three different services, you get a unified platform capable of ingesting, enriching, and analyzing data on the fly.

What's happening inside the platform

The heart of the platform is the Jet engine. It's responsible for building data processing pipelines. Jet can work equally well with continuous streams and static data sets, such as buckets in Amazon S3 or tables in a relational database.

The performance metrics here are interesting. A single Hazelcast node can aggregate 10 million events per second while keeping latency within 10 milliseconds. If you cluster the servers together, throughput scales up to a billion events per second.

To write queries against data streams, you don't need to dive into low-level Java APIs. The platform supports standard SQL. You can write a familiar SELECT against the incoming data stream, join it with an in-memory table, and immediately route the result to the target service.

Here's what connecting external sources looks like. Out of the box, you get a set of connectors:

• Apache Kafka and JMS for working with queues
• Hadoop and Amazon S3 for accessing file storage
• Relational databases via standard JDBC
• Python models for running machine learning directly inside the pipeline

Distributed memory and coordination

If you remove stream analytics from the equation, Hazelcast remains a distributed key-value store. Data is spread across cluster nodes as partitions. Developers have access to familiar Java structures (IMap, IQueue, ITopic), with the only difference being that they're distributed across the network. Point lookups by key take microseconds.

For database operations, classic caching patterns are supported: read-through, write-through, and write-behind. When using write-behind, the application saves data exclusively to Hazelcast's RAM, and the platform asynchronously flushes it to disk and the main database. If the relational DBMS temporarily goes down, your application will continue accepting requests without failures.

A separate feature is microservice coordination. Hazelcast can manage distributed locks, issue unique ID sequences, and maintain shared counters. This eliminates the need to deploy and maintain a separate Apache ZooKeeper cluster for mundane synchronization tasks.

How to build and run the project

The project's source code is written in Java. JDK 17 or newer is required for building from source. The easiest way to build the project is to use the Maven Wrapper script:

git pull origin master
./mvnw clean package -DskipTests

The full build with all checks can take a while. If you just need to quickly verify local changes, pass the -Dquick flag:

./mvnw clean package -DskipTests -Dquick

This parameter disables Javadoc generation, Checkstyle checks, and builds of secondary modules.

The testing situation is interesting. The repository has thousands of tests divided into three profiles:

• The standard ./mvnw test profile runs fast integration tests.
• The nightly ./mvnw test -P nightly-build profile includes slow tests that cannot be run in parallel.
• The full ./mvnw test -P all-tests profile sequentially runs absolutely all checks using the network.

Some tests depend on Docker. If Docker is not installed on your machine, those tests will fail. To disable them, use the -Dhazelcast.disable.docker.tests parameter. When creating a Pull Request, the project's CI server runs the full suite, so locally it's enough to run tests for your own module only.

You can write clients not only in Java. The community and company maintain official libraries for Python, Node.js, .NET, C++, and Go.

License and a couple of practical nuances

The code in the repository is split into two parts. The core is distributed under the permissive Apache License 2.0. However, some enterprise features and modules are protected by the Hazelcast Community License. It prohibits using the code to create paid managed services (Cloud Service Provider) that compete with the company's original cloud product.

The second point is resource requirements. Since all hot data resides in RAM, you'll need to purchase a substantial amount of memory to work with large volumes. Also, in a Java environment, you should pay close attention to Garbage Collector settings to avoid pauses when cleaning up gigabytes of memory. However, Hazelcast engineers mitigate this problem with off-heap storage, moving data outside the Java heap.

Who should take a look at Hazelcast

The platform performs well where real-time event response matters:

• Fraud prevention and scoring in fintech
• Processing telemetry and high-frequency IoT signals
• Calculating prices and discounts in e-commerce right at the moment of a customer's click
• Synchronizing data between distributed data centers (WAN replication)

If you only need a simple cache for a couple of endpoints, Hazelcast would be overkill: a simpler Redis would handle that task just fine. But if your project has grown to the scale where stream analytics must intersect with distributed memory without constant trips to disk storage, Hazelcast will save you months of development.