AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required

Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI • Ollama: Run LLMs Locally with One Command

What Is AirLLM? #

AirLLM dramatically reduces LLM inference memory usage, letting a 70B model run on a single 4GB GPU — without quantization, distillation, or pruning. Per the current README, that same approach scales further: Llama 3.1 405B on 8GB, DeepSeek-V3 (671B) on ~12GB, and, as of July 2026, Kimi K3 (2.8T parameters, the largest open-source model released to date) on under 4GB, because sparse MoE models let AirLLM stream one expert at a time instead of a whole dense layer.

🔗 GitHub: https://github.com/lyogavin/airllm 📦 PyPI: https://pypi.org/project/airllm

Apache-2.0 licensed, at 25,300+ GitHub stars, with a version history running continuously from its November 2023 initial release through a July 2026 commit adding Kimi K3 support — this is a mature project with a genuine multi-year track record, not a recent trend-chaser.

The Trick: One Layer on the GPU at a Time #

Most “run a huge model on small hardware” techniques (quantization, pruning, distillation) change the model itself to make it smaller. AirLLM instead changes how much of the model is ever loaded at once: it decomposes the model layer-by-layer, and during inference only ever keeps one layer on the GPU — for MoE models, one routed expert — streaming the rest from disk/CPU as needed.

The practical consequence: your required VRAM is driven by the size of the model’s largest single layer, not its total parameter count.

Same one line of code (AutoModel.from_pretrained(...)) for every row in that table — no per-model special-casing.

Quickstart #

Install #

pip install airllm

Run inference #

from airllm import AutoModel

MAX_LENGTH = 128
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")     # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

input_text = ['What is the capital of United States?']

input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

Disk space matters here: on first run, the original model is decomposed and saved layer-wise, so you need enough free disk space in your Hugging Face cache directory for both the original download and the split version — this is the single most common source of errors (see FAQ below).

Optional: Model Compression for 3x Faster Inference #

Separately from the core no-quantization layer-streaming approach, AirLLM offers opt-in block-wise quantization compression for up to 3x faster inference with what the maintainers describe as “almost ignorable accuracy loss” (methodology reference: arXiv:2212.09720).

pip install -U bitsandbytes
pip install -U airllm

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
                     compression='4bit'  # or '8bit'
                    )

Why this differs from standard quantization: typical quantization needs both weights and activations quantized to meaningfully speed things up, which makes accuracy harder to preserve. Since AirLLM’s bottleneck is disk loading rather than compute, it only needs the loading size reduced — so it quantizes weights alone, which is easier to keep accurate.

Configuration Options #

Running on macOS #

1. Install mlx and torch
2. Only Apple Silicon is supported (not Intel Macs)
3. Run the same AutoModel.from_pretrained(...) code as on Linux

The README notes you may need a native (non-Rosetta) Python installation for this to work correctly — see the linked macOS example notebook for the full setup.

Beyond Llama: Broad Model Family Support #

AirLLM works out of the box with most popular open models — pass the Hugging Face repo ID and AutoModel handles the rest:

# ChatGLM
model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base")

# Qwen
model = AutoModel.from_pretrained("Qwen/Qwen-7B")

# Baichuan / InternLM / Mistral
model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base")
#model = AutoModel.from_pretrained("internlm/internlm-20b")
#model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

Documented families: Llama (2/3/3.1/3.3/4) · Qwen (1/2/2.5/3, including MoE + FP8) · DeepSeek (V2/V3/R1) · Mistral & Mixtral · Phi · Gemma · ChatGLM · Baichuan · InternLM · Yi — plus, per the July 2026 update, Kimi K3, which additionally requires pip install compressed-tensors flash-attn, a CUDA 12 torch build (no prebuilt flash-attn wheel exists yet for CUDA 13), and transformers 4.56.x specifically (its remote code doesn’t load on the 5.x line).

AirLLM vs. Standard Approaches to Running Huge Models #

The distinguishing case for AirLLM specifically: you have exactly one GPU with limited VRAM and want to run a model that wouldn’t otherwise fit at all, without standing up multi-GPU infrastructure or accepting a quantized/distilled model as your baseline.

Use Cases #

1. Running SOTA-Scale Models on a Single Consumer or Hobbyist GPU #

The headline case: DeepSeek-V3 (671B) on ~12GB, or Llama 3.1 405B on 8GB — hardware most individual developers or small teams can actually own, not a multi-GPU server rack.

2. Evaluating a Huge Model Before Committing to Serving Infrastructure #

Since it’s a one-line AutoModel.from_pretrained() swap, AirLLM is a fast way to sanity-check a large model’s output quality on your own hardware before investing in the multi-GPU setup a production deployment would need.

3. MoE Models on Minimal Hardware #

Kimi K3 (2.8T params) running in under 4GB is only possible because of per-expert streaming — for large MoE models specifically, AirLLM’s approach scales even better than for dense models of similar size.

4. Apple Silicon Local Inference #

The macOS path (via mlx) extends the same low-VRAM approach to Apple Silicon Macs, for developers who don’t have a dedicated NVIDIA GPU at all.

Related Repositories #

Related Articles #

• Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI — for comparing AirLLM’s single-tiny-GPU niche against general self-hosting options
• Ollama: Run LLMs Locally with One Command — the simpler default when your model already fits in VRAM

Conclusion #

AirLLM solves a specific, well-defined problem: running a model that’s simply too large for your GPU’s VRAM, without touching the model’s weights by default and without needing multiple GPUs. The layer-and-expert streaming approach trades inference speed for the ability to run models — up to 671B dense-equivalent and beyond for MoE — on hardware that couldn’t otherwise load them at all, with an optional 3x compression mode when speed matters more than exact full-precision output. Nearly three years of continuous version history, current through July 2026’s Kimi K3 support, backs it as a maintained tool rather than an abandoned proof of concept.

Best for: Developers with exactly one GPU and limited VRAM who want to run or evaluate a large open model without standing up multi-GPU infrastructure or committing to a quantized/distilled variant.

GitHub: https://github.com/lyogavin/airllm

Last updated: 2026-08-02

References & Sources #

• AirLLM on GitHub
• AirLLM on PyPI
• Block-wise quantization paper (arXiv:2212.09720)
• bitsandbytes