Training Neural Networks Directly on Apple Neural Engine Without CoreML and GPU

Every Apple Silicon processor contains a dedicated block for neural network operations — the Apple Neural Engine (ANE). The manufacturer claims tens of teraflops of performance, but official tools limit its use to inference only via CoreML. Training models is offered either on CPU or the graphics chip.

A developer under the nickname maderix decided to test whether ANE hardware is physically incapable of backpropagation, or if the issue is solely due to Apple's software limitations. Over a weekend, he dissected macOS private frameworks and got ANE to run not only forward but also backward pass for transformers.

The project gathered over 7 thousand stars on GitHub. Let's break down how this hack works, what tricks had to be applied, and what the actual performance looks like.

Why Dig Into Closed Frameworks

The official stack like CoreML creates the impression that ANE is a closed black box. You give it a ready-made model, it outputs the result. If you need to train even a small model directly on the client, you had to turn to Metal or the MLX framework, loading the GPU.

The author of the ANE repository demonstrated that the chip is perfectly capable of executing arbitrary computational graphs. To do this, he reverse-engineered the private libraries _ANEClient and _ANECompiler, as well as the internal model description language MIL (Model Intermediate Language).

The result turned out interesting: full-fledged training of transformers without a single line of CoreML or Metal. The model text is assembled directly in RAM, compiled on the fly, and sent to the neural processor.

Limitations and Harsh Reality

Before rushing to rewrite your training scripts on Mac, it's worth looking at honest numbers. The author himself warns upfront: this is an academic experiment, not a production-ready library.

Achieving 100% utilization of ANE resources hasn't been possible yet. Real chip utilization is around 5–9% of peak. Software and hardware limitations make themselves known:

• Some mathematical operations are not supported by the chip in the required form and fall back to CPU.
• The backward pass for weights (dW) still has to be computed by the processor.
• ANE compiler memory leaks, so after about a hundred iterations, workarounds are needed.

Nevertheless, even in this mode, the project delivers decent speed on basic architectures.

How the Training Pipeline Works

The project architecture relies on distributing tasks between ANE and CPU. The neural network accelerator handles the heaviest matrix multiplications, while the processor takes care of the surrounding logic and gradient accumulation.

The forward pass and input gradient computation (dx) are fully executed on ANE. Weight gradients (dW) are calculated by CPU through optimized Accelerate and cblas_sgemm libraries. The Adam optimizer and RMSNorm layer also run on the processor.

To avoid recompiling the model graph at each step when weights change, the author applied a trick: weights and activations are packed into a single tensor through spatial dimensions, and inside the MIL kernel they are simply split back apart.

IOSurface memory is used for data exchange between CPU and ANE. This makes it possible to transfer tensors without unnecessary copying between address spaces. Data is packed into the ANE-specific format [1, C, 1, S], where channels come first. This approach eliminated transpose matrix overhead.

Non-Obvious Problems and Workarounds

Many Apple hardware pitfalls surfaced during reverse engineering.

First, the SDPA (Scaled Dot-Product Attention) operation in ANE ignores the causal mask attn_mask at the hardware level. The attention mechanism had to be split into three stages: multiplying Q and K on ANE, masking with softmax on CPU, and final multiplication by V back on ANE.

Second, the built-in compiler _ANECompiler contains a memory leak. After approximately 119 compilations, the process crashes due to resource exhaustion. The author solved the problem radically: when the counter approaches the limit, the program saves a checkpoint and does exec() — restarts itself with state preservation.

Third, FP16 computations during backward pass quickly lead to underflow, causing gradients to turn into zeros. The problem was solved by scaling the loss with coefficient 256 * NLAYERS.

Performance on M4

On the Apple M4 chip, the results turned out quite illustrative. Testing was conducted on two architectures:

For the Stories110M model with 109 million parameters (12 layers, classic Multi-Head Attention), the time for one training step was 91 milliseconds.

The larger Qwen3-0.6B with 596 million parameters and Grouped-Query Attention processes one step in 412 milliseconds.

The author also tested INT8 W8A8 quantization. Using 8-bit weights and activations reduces the load on the chip's L2 SRAM memory and increases throughput from 18.6 TOPS to 35.1 TOPS on M4 — a speedup of nearly 1.88x compared to FP16.

How to Run the Project

The project requires no external dependencies like PyTorch or Conda. All you need is the latest macOS 15 on a machine with Apple Silicon and the Clang compiler. Private APIs are pulled in at runtime via objc_msgSend.

To build the dynamic pipeline, simply navigate to the project folder and run the make command:

cd training/training_dynamic
make MODEL=stories110m
./train --scratch

If you want to check out INT8 quantization or benchmark your chip's peak TOPS, there are separate benchmarks in the root folder of the repository.

Final Thoughts

The maderix project is an example of quality research "under the hood" of Apple hardware. It demonstrates that ANE limitations lie solely in the software plane and the closed nature of the ecosystem.

Using the repository for training large language models in production is currently pointless — GPU and the MLX framework exist for that. But if you're studying how neural accelerators work, writing your own compilers for Edge AI, or want to understand how to work with macOS private APIs directly from C and Objective-C, this code will be an excellent learning resource.