Why the NSA Built Its Own HBase and How Apache Accumulo Works

When you need to store terabytes of data in a distributed fashion, Java developers usually think of Cassandra or Apache HBase. Few people know that HBase has an older relative with a rare feature that makes it beloved in specific projects.

In 2008, the U.S. National Security Agency faced a challenge: they needed to store colossal amounts of information but grant access to different cells of the same table strictly based on employee clearance levels. Existing NoSQL solutions couldn't do this. As a result, the intelligence agency wrote its own storage system, and in 2011 donated it to the Apache Software Foundation. The project was named Apache Accumulo.

Under the Hood

At its core, Accumulo is a sorted distributed key-value database. It was inspired by the same well-known Google paper about BigTable.

Data isn't stored on a single disk. As its foundation, Accumulo uses Apache Hadoop HDFS for physical file storage and Apache ZooKeeper for cluster state management and coordination.

If you look at the key structure in Accumulo, you'll understand how it differs from simple stores:

• Row ID (row identifier)
• Column Family (column family)
• Column Qualifier (column qualifier)
• Column Visibility (security label)
• Timestamp (timestamp)

It's the fourth component, Column Visibility, that solves the problem the project was created for in the first place.

Cell-Level Access Control

In most databases, permissions are granted at the table or column level. In extreme cases — individual rows. In Accumulo, the security context is embedded directly into each cell.

Each record contains a label like (ADMIN&SECRET)|TOP_SECRET. When a client sends a read request, it passes its authorization tokens. The Accumulo server filters the data stream itself before sending it over the network. If a user doesn't have the required clearance, they won't even know about the existence of specific cells in a row.

This approach eliminates the need to write complex filtering logic in application code or deploy dozens of separate tables for each access level.

Iterators: Server-Side Computations

The second strong suit of Accumulo is iterators. These are server-side Java plugins that are embedded into the data scanning and compaction chain.

Iterators run directly on storage nodes (Tablet Servers). They perform several tasks at once:

1. Filter data by complex conditions before sending it to the client.
2. Aggregate and transform values on the fly.
3. Clean up stale versions or deleted records during background file compaction.

Using iterators, you can make heavy analytical filtering happen right at the storage layer. This way, the network doesn't get clogged with raw gigabytes.

How to Build and Run

The project is entirely written in Java. It's built with standard Maven. The command to build the tarball looks like this:

mvn package -DskipTests

The ready archive will appear at assemble/target/accumulo-<version>-bin.tar.gz.

For local development, you can spin up a test instance, but for full-fledged operation you'll need to set up a working HDFS cluster and ZooKeeper. You'll need to allocate resources and configure the settings properly, since the system is designed for a distributed environment.

Where It Comes in Handy

There's no point in deploying Accumulo for a simple pet project. But the system shows its best side in the following scenarios:

• Multi-tenant services. When one database stores data from different departments or clients with overlapping permissions.
• Graph processing and search index building. Iterators help quickly intersect sets on the server side.
• Storing logs and telemetry events. When writes come as a continuous stream, but reads must be strictly restricted by user roles.

Is It Worth the Effort

The project has around 11,000 stars on GitHub, but behind the modest popularity lies a mature tool with frequent releases and Apache Foundation support.

If your system doesn't need security labels at the individual cell level or specific server-side processing through iterators, it's easier to choose HBase or Cassandra. They have larger communities and simpler integration with popular frameworks. However, if data security at the cell level is a top priority, there are practically no alternatives to Accumulo.