Run a 70B LLM on a 4GB GPU: Layer-by-Layer Inference with AirLLM (2026)

The usual advice for local language models is simple: buy more VRAM. A 70B model in 16-bit precision needs roughly 140 GB, so the advice sounds final.

It is not. AirLLM runs a 70B model on a single 4 GB GPU. It does this without quantization, distillation, or pruning. This tutorial explains the mechanism and shows the code. It also states the costs honestly.

This tutorial covers a software technique, not a hardware purchase. Do you want to choose a GPU and build a machine? Read the hardware build guide instead. Read this page when your hardware is fixed and the model does not fit.

Prerequisites

• A Linux or Windows machine with an NVIDIA GPU. 4 GB of VRAM is enough.
• Python 3.9 or later.
• A large amount of free disk space. The section on disk cost explains the requirement.
• Basic Python experience.

Apple Silicon users need MLX instead. AirLLM supports macOS only on Apple Silicon hardware.

1. Why a 70B Model Normally Needs 140 GB

A model stores its weights as numbers. The memory it needs follows from the count of those numbers and the size of each one.

A 70-billion-parameter model in 16-bit precision uses two bytes per parameter. The arithmetic gives roughly 140 GB. No consumer GPU has that much memory. The largest consumer cards stop at 32 GB.

The standard answers each cost something:

AirLLM takes a different route and keeps the full-precision weights.

2. The Mechanism: One Layer at a Time

A transformer model is a stack of layers. Inference runs through them in order. Layer 1 produces a result, layer 2 consumes it, and so on to the end.

Here is the insight: the GPU needs only the current layer. Layer 40 does not need layer 3 in memory anymore.

AirLLM keeps one layer on the GPU at a time. It loads a layer, runs it, discards it, and loads the next. Peak VRAM use therefore depends on the size of the largest single layer, not on the size of the whole model.

The project reports these requirements:

Sparse Mixture-of-Experts models gain even more. AirLLM streams one expert at a time rather than a whole layer, so a 671B MoE model fits in about 12 GB.

3. Install AirLLM

Install the package with pip:

pip install airllm

Use a virtual environment to keep the install separate from your system Python:

python3 -m venv airllm-env
source airllm-env/bin/activate
pip install airllm

4. Run Your First Model

The API mirrors the Hugging Face style. Give it a model ID, and it handles the rest:

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_tokens = model.tokenizer(
    ['What is the capital of United States?'],
    return_tensors="pt",
    truncation=True,
    max_length=128,
)

output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
)

print(model.tokenizer.decode(output.sequences[0]))

The first run downloads the model and splits it into per-layer files. This step takes a long time. Later runs reuse the split files and start faster.

AirLLM supports most popular open models without model-specific code. The list includes Llama 2, Llama 3, Llama 3.1, Llama 4, Qwen, DeepSeek, Mistral, Phi, and Gemma.

5. The Real Costs

The VRAM benefit is genuine. It is not free.

Speed is the main cost. The GPU loads every layer from disk for every token that it generates. Disk speed, not GPU speed, sets the pace. Expect an experience closer to a batch job than a chat.

Disk space is the second cost. AirLLM decomposes the model into layer files, and that step needs substantial free space. Check your free space before you start a 405B model.

Optional compression trades accuracy for speed. AirLLM supports 4-bit and 8-bit block compression, and the project reports about a 3x inference speedup with it. This step reintroduces quantization, so use it only when the speed matters more than the precision.

6. When to Choose Something Else

AirLLM solves one problem well: a model that does not fit, on hardware you cannot change. Other tools fit other problems better.

• Your model nearly fits. Use llama.cpp with partial GPU offload. It keeps most layers resident and stays far faster.
• You accept quantization. A 4-bit 70B model needs roughly 40 GB and runs at normal speed on two 24 GB cards. Ollama makes this straightforward.
• You need interactive latency. No layer-streaming approach will feel like a chat. Choose a smaller model instead.
• You need throughput for many users. Use a served model behind an API.

Pick AirLLM when you must run a specific large model, the hardware is fixed, and you can wait.

7. A Practical Workflow

Layer streaming suits batch work. This pattern fits the tool:

1. Collect your prompts into a list.
2. Start the job and let it run unattended.
3. Write each result to disk as it completes.
4. Review the output later.

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

prompts = [
    "Summarise the CAP theorem in two sentences.",
    "Explain the difference between TCP and UDP.",
]

with open("results.txt", "w") as handle:
    for prompt in prompts:
        tokens = model.tokenizer(
            [prompt], return_tensors="pt", truncation=True, max_length=128
        )
        output = model.generate(tokens['input_ids'].cuda(), max_new_tokens=128)
        handle.write(model.tokenizer.decode(output.sequences[0]))
        handle.write("\n---\n")

The script writes each answer as soon as the model produces it. A crash therefore costs you one prompt, not the whole run.

Quick Reference

Sources

• AirLLM repository and README — mechanism, VRAM figures, supported models, and compression claims

Related Tutorials

• Building a Local LLM Machine in 2026
• Best GPU for Hashcat 2026
• Structured output from LLMs in Python