Modal Serverless GPU Compute — Run ML Pipelines Without Infrastructure

TL;DR #

Modal is a Python-native serverless compute platform that lets you run GPU-accelerated workloads without managing any infrastructure. You write standard Python functions, decorate them with @modal.enter() and @modal.function(), and Modal handles container provisioning, GPU allocation, networking, and scaling. Perfect for LLM inference endpoints, fine-tuning jobs, and batch ML pipelines.

What Is Modal? #

Modal is a serverless compute platform designed specifically for machine learning and data-intensive workloads. Unlike traditional cloud providers where you provision VMs, manage Kubernetes clusters, or configure auto-scaling groups, Modal abstracts all infrastructure away into simple Python decorators.

The core philosophy is simple: your code is your infrastructure definition. Write a Python function, add a few decorators specifying resource requirements (GPU type, memory, timeout), and deploy. Modal automatically provisions the right containers, scales them based on incoming requests, and bills you per-second of actual compute used.

Why Serverless GPU Matters for AI #

GPU infrastructure has historically been the biggest bottleneck in AI development. Traditional approaches require:

• Pre-provisioning GPU instances (expensive idle time)
• Managing Kubernetes clusters for orchestration (complex ops overhead)
• Handling cold starts for inference endpoints (latency issues)
• Scaling from zero to thousands of concurrent requests (manual tuning)

Modal solves all these problems by treating GPUs as a first-class serverless primitive. You pay only for the seconds your GPU is actually running inference or training, with no minimum commitment.

import modal

# Define a container image with PyTorch and CUDA pre-installed
stub = modal.Stub("my-modal-app")

image = modal.Image.debian_slim().pip_install(
    "torch",
    "transformers",
    "accelerate"
)

Key Differentiators vs Alternatives #

*Warm pools reduce cold start to <2 seconds for most models.

Getting Started: Your First Modal App #

Step 1: Install and Authenticate #

# Install the Modal Python SDK
pip install modal-client

# Authenticate with your Modal account
modal setup

Modal provides free tier credits for new accounts — typically enough to run several hours of A10G compute for testing.

Step 2: Write a Simple Inference Function #

import modal
from transformers import AutoModelForCausalLM, AutoTokenizer

stub = modal.Stub("llm-inference")

# Pre-load model once at container startup
@stub.cls(
    image=modal.Image.debian_slim().pip_install("transformers", "torch", "accelerate"),
    gpu="A10G",
    memory=8192
)
class LLMEndpoint:
    @modal.enter()
    def load_model(self):
        self.model = AutoModelForCausalLM.from_pretrained(
            "meta-llama/Llama-3.2-3B-Instruct",
            torch_dtype="auto",
            device_map="auto"
        )
        self.tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

    @modal.method()
    def generate(self, prompt: str, max_tokens: int = 512) -> str:
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        outputs = self.model.generate(**inputs, max_new_tokens=max_tokens)
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

This class-based approach keeps the model loaded in memory across requests, eliminating the multi-minute cold start penalty that plagues serverless LLM deployments.

Step 3: Deploy and Test #

# Deploy the app to Modal cloud
modal deploy my_app.py

# Test from the command line
modal run my_app::LLMEndpoint.generate --prompt "Explain quantum computing" --max_tokens 256

After deployment, Modal assigns your endpoint a public URL. Any client can call it via HTTP REST API.

Deployment Patterns #

Pattern 1: High-Throughput Inference Endpoint #

For production LLM serving, use Modal’s built-in concurrency and request queuing:

@stub.cls(
    gpu="L4",
    concurrency_limit=20,
    allow_concurrent_inputs=10,
    keep_warm=2  # Keep at least 2 containers warm
)
class ProductionLLM:
    @modal.enter()
    def load_model(self):
        self.model = load_optimized_model()  # Your optimization here
        self.tokenizer = AutoTokenizer.from_pretrained("your-model")

    @modal.web_endpoint(method="POST")
    def infer(self, req: dict):
        prompt = req.get("prompt", "")
        result = self.model.generate(prompt, max_tokens=req.get("max_tokens", 256))
        return {"response": result}

Key settings:

• keep_warm=2: Ensures 2 containers stay hot to handle burst traffic
• allow_concurrent_inputs=10: Each container handles 10 simultaneous requests
• concurrency_limit=20: Maximum 20 containers total (cost control)

Pattern 2: Batch Processing Pipeline #

For processing thousands of documents through an LLM:

@stub.function(
    image=image,
    gpu="A100-80GB",
    timeout=3600,  # 1 hour max
    retries=2
)
def batch_embed(docs: list[str]) -> list[list[float]]:
    """Process a batch of documents and return embeddings."""
    model = get_embedding_model()
    return model.encode(docs, batch_size=64).tolist()

# Run batch job
results = batch_embed.remote([f"Document {i}" for i in range(10000)])

Modal handles chunking, retrying failed batches, and parallelizing across multiple GPU containers automatically.

Pattern 3: Fine-Tuning Job #

@stub.function(
    gpu="H100-80GB",
    memory=16384,
    timeout=14400  # 4 hours
)
def run_finetune(dataset_path: str, output_dir: str):
    """Run LoRA fine-tuning on a dataset."""
    from trl import SFTTrainer
    from peft import LoraConfig

    model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")
    tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

    peft_config = LoraConfig(
        r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],
        lora_dropout=0.05, bias="none", task_type="CAUSAL_LM"
    )

    trainer = SFTTrainer(
        model=model, tokenizer=tokenizer,
        train_dataset=load_dataset(dataset_path),
        peft_config=peft_config,
        args=TrainingArguments(output_dir=output_dir, num_train_epochs=3)
    )
    trainer.train()
    trainer.save_model(output_dir)

Deploy with modal run finetune.py --dataset_path s3://my-bucket/data --output_dir /mnt/output. Modal mounts the output directory to persistent storage.

Pricing and Cost Optimization #

Understanding Modal’s Pricing Model #

Modal charges based on the resources your containers actually use:

These are approximate; check modal.com/pricing for current rates.

Cost Optimization Strategies #

Strategy 1: Right-size GPU selection

# Don't use H100 for a 3B parameter model
# Use A10G instead — saves 75% cost
@stub.function(gpu="A10G", memory=4096)
def light_inference(prompt: str):
    model = load_small_model()  # 3B params fits easily
    return model.generate(prompt)

# Reserve H100 only for large-scale fine-tuning
@stub.function(gpu="H100-80GB", memory=32768)
def heavy_finetune(config: dict):
    return run_large_scale_training(config)

Strategy 2: Use keep_warm strategically

# For predictable traffic: keep warm during business hours only
@stub.function(gpu="L4", keep_warm=1)
def production_endpoint():
    ...

# For bursty traffic: use higher concurrency_limit
@stub.function(gpu="L4", concurrency_limit=50, keep_warm=3)
def bursty_endpoint():
    ...

Strategy 3: Container reuse with @stub.cls

Class-based functions keep state in memory, avoiding repeated model loading. This is critical for LLM workloads where loading a model takes 2-5 minutes.

# ❌ Bad: loads model on every invocation
@stub.function(gpu="A10G")
def bad_approach(prompt: str):
    model = load_model()  # Reloads every call!
    return model.generate(prompt)

# ✅ Good: loads model once, reuses across requests
@stub.cls(gpu="A10G")
class GoodApproach:
    @modal.enter()
    def setup(self):
        self.model = load_model()  # Loads once at startup
    
    @modal.method()
    def generate(self, prompt: str):
        return self.model.generate(prompt)  # Reuses loaded model

Real-World Cost Comparison #

Advanced Features #

Secret Management #

Never hardcode API keys. Modal’s secret manager injects credentials at runtime:

import modal

stub = modal.Stub("secret-demo")

@stub.function(
    secrets=[
        modal.Secret.from_name("huggingface-token"),
        modal.Secret.from_name("openai-key"),
    ]
)
def secure_inference(prompt: str):
    import os
    hf_token = os.environ["HF_TOKEN"]  # Injected from secret
    openai_key = os.environ["OPENAI_API_KEY"]
    return call_api(prompt, hf_token, openai_key)

Create secrets once:

modal secret create huggingface-token HF_TOKEN=your_token_here
modal secret create openai-key OPENAI_API_KEY=sk-...

Volume Mounts for Persistent Storage #

Modal volumes provide shared, persistent filesystems across function invocations:

# Create a volume for model checkpoints
checkpoint_volume = modal.Volume.from_name("model-checkpoints", create_if_missing=True)

@stub.function(
    gpu="A100-80GB",
    volumes={"/checkpoints": checkpoint_volume},
    timeout=7200
)
def fine_tune_and_save(dataset_url: str):
    # Load dataset
    dataset = load_dataset(dataset_url)
    
    # Train and save to mounted volume
    trainer.train()
    trainer.save_model("/checkpoints/final-model")
    
    print(f"Checkpoint saved to volume. Size: {os.path.getsize('/checkpoints/final-model')}")

# Access saved model from another function
@stub.function(volumes={"/checkpoints": checkpoint_volume})
def load_and_infer(prompt: str):
    model = AutoModelForCausalLM.from_pretrained("/checkpoints/final-model")
    return model.generate(prompt)

Volumes persist data across function calls, making them ideal for model checkpoints, datasets, and cache directories.

Egress Control #

Control outbound network access for security and cost management:

@stub.function(
    gpu="L4",
    network_mounts={"/etc/resolv.conf": modal.NetworkMount()},
    blocked_subnets=["169.254.0.0/16"],  # Block metadata service
    allowed_domains=["api.openai.com"]   # Only allow specific domains
)
def restricted_inference(prompt: str):
    return call_openai(prompt)

Custom Docker Images #

For complex dependencies not covered by pip_install:

custom_image = (
    modal.Image.from_dockerhub("nvidia/cuda:12.2.0-devel-ubuntu22.04")
    .apt_install("git", "cmake", "build-essential")
    .pip_install("torch", "transformers", "bitsandbytes")
    .copy_local_dir("./my-custom-model", "/app/model")
)

@stub.function(image=custom_image, gpu="A100-80GB")
def custom_model_inference(request: dict):
    model = torch.load("/app/model/best.pt")
    return model.predict(request["input"])

Troubleshooting Common Issues #

Issue 1: Container OOM Kills During Inference #

Error: Container killed due to memory limit exceeded

Fix: Increase memory allocation and enable swap:

@stub.cls(
    gpu="A100-80GB",
    memory=32768,  # 32GB RAM for large models
    ephemeral_disk=100_000  # 100GB disk for model weights
)
class LargeModel:
    @modal.enter()
    def load(self):
        self.model = AutoModel.from_pretrained(
            "big-model",
            torch_dtype=torch.float16,  # Use half precision
            device_map="auto"
        )

Issue 2: Slow Cold Starts on First Request #

Warning: First request took 180 seconds (model loading)

Fix: Use keep_warm and pre-warm containers:

@stub.cls(
    gpu="A10G",
    keep_warm=3,  # Always have 3 warm containers
    timeout=600
)
class WarmEndpoint:
    @modal.enter()
    def load(self):
        self.model = load_model()
        print("Model loaded successfully")

Issue 3: Timeout During Long Fine-Tuning Jobs #

Error: Function timed out after 3600 seconds

Fix: Increase timeout and use volumes for checkpoint saving:

@stub.function(
    gpu="H100-80GB",
    timeout=28800,  # 8 hours
    volumes={"/data": modal.Volume.from_name("training-data")}
)
def long_training_job(config_path: str):
    for epoch in range(10):
        train_epoch(config_path)
        if epoch % 2 == 0:
            save_checkpoint(f"/data/checkpoint-{epoch}")

Issue 4: Concurrency Throttling #

Error: Too many concurrent inputs (limit: 10)

Fix: Adjust concurrency settings:

@stub.cls(
    gpu="L4",
    concurrency_limit=100,       # Max containers
    allow_concurrent_inputs=20,  # Requests per container
    keep_warm=5                  # Warm pool size
)
class ScalableEndpoint:
    @modal.method()
    def handle(self, request: dict):
        return process(request)

Future Directions #

Modal’s Roadmap for 2026 #

Modal continues to invest heavily in ML infrastructure. Key upcoming features include:

1. Multi-node distributed training: Native support for training across 8+ GPUs with automatic data parallelism
2. GPU sharing: Time-slicing GPUs for better utilization during low-traffic periods
3. Custom GPU types: Support for next-gen GPUs (Blackwell B200) as they become available
4. Edge deployment: Deploy Modal functions to edge locations for sub-50ms inference latency
5. Native vector DB integration: Built-in vector search backed by Modal’s storage layer

When to Choose Modal #

Choose Modal when:

• You want to ship ML workloads in hours, not weeks
• Your workload is sporadic (batch jobs, infrequent inference)
• You need GPU access without cluster management
• Your team is Python-first and wants minimal DevOps

Consider alternatives when:

• You need absolute lowest latency (<10ms) — bare metal or dedicated instances win
• You have predictable 24/7 high throughput — reserved instances may be cheaper
• You need custom kernel modifications — Modal uses standard container images
• You’re deeply invested in a specific cloud’s ecosystem — native services may integrate better

Community Updates #

The serverless GPU space is heating up rapidly. In mid-2026, several new entrants joined the market:

• RunPod Serverless launched competitive GPU pricing starting at $0.30/hr for A10G
• Replicate expanded their model library to 500+ pre-packaged ML models
• AWS Lambda GPU announced general availability for Graviton4 + Inferentia2 combinations

Despite this competition, Modal maintains its lead in developer experience — the Python-native API means teams can go from prototype to production without learning YAML, Helm charts, or Terraform.

The community-driven model registry on Modal has grown to over 2,000 models, covering everything from LLMs to diffusion models to speech recognition. Users can browse, test, and deploy any registered model with a single line of Python code.

FAQ #

Q: How does Modal compare to running GPUs on AWS EC2 directly? #

Modal eliminates the operational overhead of managing GPU instances. On EC2, you handle spot instance interruptions, driver updates, GPU monitoring, and auto-scaling configuration. With Modal, all of this is abstracted away — you just write Python functions. For sporadic workloads, Modal is typically 70-90% cheaper because you only pay for actual compute time rather than keeping instances running 24/7.

Q: Can I use Modal with my existing Hugging Face models? #

Yes. Modal works seamlessly with Hugging Face models. Simply install the transformers library in your image and load models using the standard AutoModel.from_pretrained() API. You can also mount Hugging Face tokens as Modal secrets for private model access. Many users report loading times of 30-60 seconds for models under 10B parameters.

Q: What happens if my GPU container crashes mid-request? #

Modal automatically retries failed containers with configurable retry policies. For inference endpoints, you can set retries=3 on your function definition. For training jobs, Modal supports checkpoint-based recovery — save your checkpoint to a Modal volume, and on retry, resume from the last checkpoint rather than restarting from scratch.

Q: Is there a free tier for testing? #

Modal offers free credits for new accounts, typically sufficient for 10-20 hours of A10G compute. This is enough to prototype and test most ML workloads before committing to paid usage. No credit card required to start.

Q: How do I monitor and debug running Modal functions? #

Modal provides a web dashboard at modal.com/apps showing real-time metrics: invocation count, latency percentiles, error rates, and GPU utilization. You can also stream logs directly from the CLI with modal logs <app-name> and set up alerts for error thresholds or cost limits.

Q: Can I run Modal functions on-premises or in air-gapped environments? #

Currently, Modal operates exclusively on their managed cloud infrastructure. They do not offer an on-premises deployment option. For air-gapped environments, consider alternatives like vLLM with Kubernetes or Ray Serve, which can run entirely within your own infrastructure.

Sources #

• Modal Documentation
• Modal GitHub Examples
• Modal Pricing Page
• Serverless GPU Computing Survey — ACM Queue 2026
• Comparing Cloud GPU Costs — ML Infrastructure Report Q2 2026

Join our Telegram Group for real-time AI tool discussions and deployment tips: t.me/dibi8