How to speed up data compression fourfold without rewriting code

Why change something that has worked since the nineties

The original zlib library appeared thirty years ago. Mark Adler and Jean-Loup Gailly created an incredibly reliable and portable tool. It works almost everywhere, from smartwatches to legacy operating systems. But this universality comes with a downside.

To maintain compatibility with old compilers and 16-bit environments, zlib code has to drag along a bunch of workarounds. It's full of memory limit checks and macros for supporting archaic processors. For this reason, fresh optimizations for vector instructions by modern developers simply couldn't be pushed into the main repository. Mark Adler maintains a high bar for stability, and his approach is understandable.

That's when Hans Christian Rosbach (Dead2) decided to compile the accumulated community patches into a single fork. That's how the zlib-ng project was born.

What changed inside zlib-ng

The fork's idea is simple: drop the ancient legacy code and apply vector instructions of modern CPUs. Developers took patches from Intel and Cloudflare, cleaned out examples from the nineties, and rewrote key algorithms in C11.

The results were tangible. On x86-64 architecture, compression and decompression work approximately four times faster than standard zlib.

What drives this performance gain:

• Vector instructions for different architectures. AVX2, AVX-512, SSSE3 for x86, NEON for ARM, as well as vector units for POWER, RISC-V, LoongArch, and IBM Z are utilized.
• Automatic CPU capabilities detection at runtime. The compiled binary itself knows which vectorized functions to call on a specific processor.
• Fast deflate algorithms. Intel's work on optimizing repeat search, hash shifting, and CRC32 computation is used.
• Safe unaligned access. Reading and writing of unaligned memory is optimized along with an updated bit buffer.

At the same time, the project doesn't try to break the existing world. The library can be built with an API fully compatible with standard zlib, or you can use its own updated native API.

How to build and try it yourself

The project has two build systems: CMake and good old configure.

If you use CMake, the build looks familiar:

Parameter enables full compatibility mode with classic zlib. The output is a library that can be substituted for the system .

For fans of the familiar build process is no different:

If you use the vcpkg dependency manager, you won't need to pull sources manually at all:

Speeding up applications with LD_PRELOAD

One of the most interesting tricks with zlib-ng is speeding up existing software without changing its code. If your program links with dynamically, you can substitute the library at runtime.

Here's how to launch a utility with a temporary substitution:

The application immediately utilizes your processor's vector instructions without any changes to the binary.

However, the fork's authors specifically warn in the README: don't try to replace the system at the distribution level in Linux directories. If something goes wrong or a rare incompatibility surfaces, the entire system will crash, including system services. It's safer to install the fork in a separate directory like and link it explicitly.

How well is the code covered by tests

Code responsible for data compression must work without failures. Losing even a single bit will turn an archive into garbage. The zlib-ng authors took testing seriously.

The repository employs a comprehensive set of checks:

• Memory sanitizers and fuzzing via OSS-Fuzz.
• Native and emulated CI via QEMU for ARM, PowerPC, RISC-V, SPARC64, and S390x.
• Unit tests based on Google Test.
• Performance measurements using Google Benchmark.

Thanks to continuous fuzzing, the library maintains a high level of reliability, which is critical when replacing core system components.

Who will benefit from zlib-ng

First and foremost, the fork will be useful for teams where compression and decompression of data have become a bottleneck. If you're processing gigabytes of logs, working with web servers like nginx, compressing textures in game development, or processing enormous amounts of data in the backend, zlib-ng will deliver a noticeable speed boost.

On the other hand, if your software runs on ancient 16-bit microcontrollers or completely exotic three-decade-old operating systems, there's no point in touching the proven original zlib. The fork was created specifically for modern platforms.

The tool looks mature and is actively maintained by the community. If you need maximum DEFLATE and gzip performance on modern hardware, it's worth spending a couple of hours and running benchmarks on your data.