Exploring LitGPT - How to Fine-Tune and Run Open Neural Networks Without Heavy Frameworks

If you've ever tried to dive into Hugging Face transformers source code to fix attention logic or see how a specific Llama layer is structured, you probably remember that feeling. Dozens of abstractions, twisted inheritance, megabytes of wrappers. In the end, simple debugging turns into a detective story.

Developers from Lightning AI took a different approach. They took over 20 popular language model architectures and rewrote them from scratch in pure PyTorch. The project is called LitGPT. There's no spaghetti of abstractions here. Each model fits into understandable files that are easy to read and modify.

What the Project Is About

Among the supported networks are Llama 3, Qwen 2.5, Phi 4, Gemma 2, DeepSeek R1 Distill, and hybrid MoE. The tool provides a complete pipeline: from fine-tuning and pre-training to local deployment as a REST API and quality evaluation on benchmarks like MMLU or TruthfulQA.

Everything works directly from the command line or through a minimalist Python API.

from litgpt import LLM

llm = LLM.load("microsoft/phi-2")
text = llm.generate("Fix the spelling: Every fall, the family goes to the mountains.")
print(text)

No sprawling boilerplate with dataloaders and tokenizers. The invocation code takes literally three lines.

Practical Benefits and Features

First, open source code without custom framework wrappers. If you want to understand how Flash Attention v2 or QLoRA actually work, LitGPT code serves as an excellent textbook. Each model is implemented as transparently as possible.

Second, modest hardware requirements. Out of the box, it supports 4-bit and 8-bit quantization, bitsandbytes integration, FSDP (Fully Sharded Data Parallel) for distribution across multiple GPUs, as well as CPU offloading. This makes it possible to run or fine-tune models even on a single relatively affordable graphics card.

Third, flexible ready-made training recipes in YAML format. You can run fine-tuning with LoRA or QLoRA with a single command line.

Command Line and Use Cases

The CLI in LitGPT covers most typical ML engineer tasks. For example, if you have a JSON data file and want to adapt a model to corporate documents or a specific dataset, the entire process fits into three steps.

Launching fine-tuning:

litgpt finetune microsoft/phi-2 \
  --data JSON \
  --data.json_path my_custom_dataset.json \
  --data.val_split_fraction 0.1 \
  --out_dir out/custom-model

After training completes, you can check the resulting model's responses right in the interactive terminal chat:

litgpt chat out/custom-model/final

And when the model is ready for deployment, spinning up an HTTP server takes literally a second:

litgpt serve out/custom-model/final

The result is a web server based on FastAPI with an endpoint at /predict. Send a POST request with a prompt there and get a response. This eliminates the need to build your own service with Triton or vLLM when you need a quick prototype or internal tool for the team.

Technical Configurations

When standard command line parameters become insufficient, YAML configs come to the rescue. The config_hub folder of the repository contains curated recipes for specific models. They configure quantization, micro-batch size, LoRA parameters (rank, alpha, dropout), learning rate, and optimizer schedule.

Any value from the YAML file can be overridden directly in the CLI at launch, which is convenient for quick hyperparameter sweeping:

litgpt finetune \
  --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml \
  --lora_r 4

Where the Tool Has Proven Itself

The LitGPT project long ago moved beyond a utility and became a foundation for real research.

For example, the well-known compact TinyLlama model with 1.1 billion parameters was trained using LitGPT code. The MicroLlama project authors (a 300M parameter model) also used this codebase. Researchers from Microsoft used LitGPT as the foundation for the Samba project, where they combined State Space Models with the Attention mechanism.

Additionally, LitGPT served as the official starter kit at the NeurIPS 2023 LLM Efficiency Challenge, where participants fine-tuned models in 24 hours on a single GPU.

Who Will Benefit and Is It Worth Trying

LitGPT will be useful in three situations.

1. You're studying the architecture of modern language models and want clean PyTorch code without external dependencies.
2. You need to quickly test a hypothesis, run QLoRA fine-tuning on a dataset, and serve the model via API without writing infrastructure code.
3. You're looking for a working starter template for your own LLM training research project.

The Apache 2.0 license removes any restrictions for commercial use. The developers actively maintain the repository, adding new architectures like Gemma 3 and Qwen 2.5 Coder as soon as they are released.