How to Speed Up Python with Rust and PyO3

Python is beautiful in its conciseness and rich selection of third-party packages, but when it comes to heavy computations, parsing gigabytes of data, or validating millions of objects, the interpreter starts to slow down. Previously, in such situations, developers had to write C API extensions manually. This process is labor-intensive: debugging memory leaks and manual reference management quickly eliminate all the joy of development.

A safer path emerged several years ago. The PyO3 library enables writing native modules in Rust while maintaining strict memory safety guarantees and clear abstractions.

If you've used the Pydantic v2 validator, the orjson JSON parser, the Polars framework, or OpenAI's Tiktoken tokenizer, you've already run code created with PyO3.

Why Link Rust and Python

Python excels at business logic and rapid prototyping. Rust wins where maximum speed, low resource consumption, and concurrency are needed.

PyO3 solves two problems:

1. Offloading Python application bottlenecks to separate Rust functions or structures.
2. Embedding the Python interpreter in a Rust binary for script execution.

Most often, the project is used for the first scenario.

How to Write a Module in Practice

For building Rust code into a Python package, the authors recommend the maturin tool. It handles compilation and creates wheel files.

Let's create a simple module that adds two numbers and returns the result as a string.

First, let's set up the virtual environment:

mkdir string_sum && cd string_sum
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3

The Rust-side logic in the src/lib.rs file:

use pyo3::prelude::*;

#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())
}

#[pymodule]
mod string_sum {
    use super::*;

    #[pymodule_export]
    fn init(m: &Bound<'_, PyModule>) -> PyResult<()> {
        m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
        Ok(())
    }
}

The #[pyfunction] and #[pymodule] macros automatically generate code for Python's C API. Developers don't need to manually convert PyObject types to Rust structures and back.

Let's start the build:

maturin develop

Now the function can be called directly from Python:

import string_sum

result = string_sum.sum_as_string(5, 20)
print(result)  # '25'

If you want to measure real performance in production, simply build the module with the optimization flag: maturin develop --release.

Real-World Use Cases

In Pydantic V2, all type checking and validation logic was rewritten in Rust in a separate package pydantic-core. Performance increased dramatically.

The Polars library is displacing Pandas when working with large datasets thanks to parallel computations in Rust. The user writes familiar Python code, while the heavy work goes into the compiled module.

In machine learning, PyO3 is used by Hugging Face in the tokenizers package and by OpenAI in tiktoken. Fast text parsing before sending to the neural network significantly reduces service latency.

Supporting Tools

A set of useful utilities has formed around PyO3:

• maturin eliminates the need to write complex configs for setup.py.
• rust-numpy binds NumPy arrays with Rust without copying data in memory.
• pyo3-async-runtimes connects asyncio with the Tokio async runtime.
• pythonize transforms data structures using Serde.

Conclusion

If your project has a performance bottleneck, PyO3 will help solve the problem without switching to C or C++. The main difficulty here lies in learning Rust itself. Understanding ownership concepts and data types will be required, but the library fully handles the routine work with CPython.