i
n
|
| llama.cpp (CLI) | Q4_K_M | ~65 tok/s | ~5 min |
| LocalAI | Q4_K_M | ~38 tok/s | ~15 min |*Source: SitePoint benchmark, March 2026. Single-stream generation, 256-token output.*### Concurrent Load (50 Users, RTX 4090)| Tool | Aggregate tok/s | p99 Latency | Architecture |
|------|----------------|-------------|--------------|
| Ollama | ~155 tok/s | ~24.7s | FIFO queue |
| vLLM | ~920 tok/s | ~2.8s | Continuous batching |
| llama.cpp server | ~140 tok/s | ~26s | FIFO queue |
| LocalAI | ~130 tok/s | ~28s | FIFO queue |*Ollama processes requests sequentially; vLLM's continuous batching provides 6x throughput under concurrent load. For single-user develo```
yam
l
services:
  ollama:
    image: ollama/ollama:rocm
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video
    environment:
      - HSA_OVERRIDE_GFX_VERSION=11.0.0
```2s
|
| vLLM | 400 MB | 5.5 GB | 5s |
| LocalAI | 400 MB | 5.5 GB | 8s |
| LM Studio | 800 MB | 5.8 GB | 5s |### Real-World Deployment Patterns1. **Individual Developer**: Ollama + Continue.dev for AI-assisted coding. Lat```
yam
l
services:
  ollama:
    image: ollama/ollama:0.6.7
    environment:
      - OLLAMA_NUM_PARALLEL=4      # 4 concurrent requests
      - OLLAMA_MAX_LOADED_MODELS=2  # Keep 2 models in VRAM
      - OLLAMA_KEEP_ALIVE=30m      # Unload after 30min idle
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
```n
Hardening### Environment Variables```
bas
h
# Core settings
OLLAMA_HOST=0.0.0.0:11434          # Bind to all interfaces
OLLAMA_KEEP_ALIVE=24h               # Keep models loaded for 24 hours
OLLAMA_NUM_PARALLEL=4               # Max concurrent requests
OLLAMA_MAX_LOADED_MODELS=2          # Max models in VRAM simultaneously
OLLAMA_FLASH_ATTENTION=1            # Enable Flash Attention (faster inference)# Performance tuning
OLLAMA_GPU_OVERHEAD=200MB           # Reserve VRAM headroom
OLLAMA_DEBUG=1                      # Verbose logging
```### Reverse Proxy with Nginx```
ngin
x
server {
    listen 443 ssl http2;
    server_na```
dockerfil
e
# Modelfile
FROM llama3.2:8b

# System prompt defines personality
SYSTEM """You are a senior software engineer. Be concise, 
practical, and always include working code examples."""

# Parameter tuning
PARAMETER temperature 0.3
PARAMETER num_ctx 16384
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1
PARAMETER stop "<|eot_id|>"

# Custom template (optional — inherits from base if omitted)
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
``` inference
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}
```### API Key Authentication (No Native Support)Ollama does not include built-in API key authentication. Add it via a reverse proxy:```
pytho
n
# ollama-auth-proxy.py (Flask example)
from flask import Flask, request, Response
import requestsapp = Flask(__name__)
OLLAMA_URL = "http://localhost:11434"
VALID_KEYS = {"sk-your-api-key-here"}@app.route('/', defaults={'path': ''}, methods=['GET', 'POST', 'PUT', 'DELETE'])
@app.route('/<path:path>', methods=['GET', 'POST', 'PUT', 'DELETE'])
def proxy(path):
    api_key = request.headers.get('Authorization', '').replace('Bearer ', '')
    if api```
bas
h
# Create the custom model
ollama create senior-dev -f Modelfile

# Run it
ollama run senior-dev

# View the effective Modelfile
ollama show senior-dev --modelfile
```        headers={k: v for k, v in request.headers if k != 'Host'},
        data=request.get_data(),
        stream=True
    )
    return Response(resp.iter_content(chunk_size=1024), status=resp.status_code```
dockerfil
e
# Modelfile.code-review
FROM codellama:7b-code

SYSTEM """You are a code review assistant. Analyze the provided code for:
1. Bugs and logic errors
2. Security vulnerabilities (SQL injection, XSS, buffer overflow)
3. Performance issues (N+1 queries, unnecessary allocations)
4. Style and readability

Format your response as:
- [CRITICAL] for bugs/security
- [WARN] for performance
- [INFO] for style suggestions

Always suggest a fix for [CRITICAL] and [WARN] items."""

PARAMETER temperature 0.1
PARAMETER num_ctx 8192
PARAMETER num_predict 2048
```, wrap the `api/ps` endpoint with a Prometheus exporter or use the [ollamaMQ](https://github.com/Chleba/ollamaMQ) proxy with built-in metrics.### Systemd Service (Linux)```
in
i
# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama LLM Service
After=network-online.target[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=24h"[Install]
WantedBy=default.target

ollama
```## Comparison with Alternatives| Feature | Ollama | llama.cpp | vLLM | LocalAI ```
dockerfil
e
# Modelfile.local
FROM ./my-fine-tuned-model-q4_k_m.gguf

PARAMETER temperature 0.7
PARAMETER num_ctx 4096

SYSTEM "You are a helpful assistant specialized in medical terminology."
```~65 (Q4) | ~71 (FP16) | ~38 (Q4) |
| **Multi-User Batching** | FIFO queue | FIFO queue | Continuous | FIFO queue |
| **50-User Aggregate** | ~155 tok/s | ~140 tok/s | ~920 tok/s | ~130 tok/s |```
bas
h
ollama create med-assistant -f Modelfile.local
```c
k
e
r
|
| **Modelfile/Dockerfile** | Yes | No | No | No |
| **OpenAI API Compatible**```
bas
h
# Show model details and Modelfile
ollama show llama3.2:8b --modelfile

# Show parameters only
ollama show llama3.2:8b --parameters

# Show system prompt
ollama show llama3.2:8b --system

# List all local models
ollama list

# Show running models
ollama ps
```a
3.1 8B on RTX 4090, March 2026. Sources: SitePoint, TowardsAI, LocalAI Master.*### When to Choose What- **Ollama**: Start here. Best developer experience, fastest setup, excellent single-user performance. Use for local development, small-team deployments, and edge devices.
- **llama.cpp**: Choose if you need maximum control over inference parameters, custom kernels, or low-level optimizations. Good for embedded systems where you compile from source.
- **vLLM**: Choose when serving 5+ concurrent users with SLA requirements. Continuous batching and PagedAttention deliver production-grade throughput that Ollama cannot match at scale.
- **LocalAI**: Choose if you need a drop-in OpenAI replacement supporting image generation (Stable Diffusion), speech-to-text (Whisper), and full API parity in a container.## Limitations / Honest Assessment**No built-in authentication.** Ollama assumes a trusted local network. For internet-facing deployments, you must add an authentication layer (reverse proxy, API gateway, or VPN). This is the most common production oversight.**No continuous batching.** Under concurrent load, Ollama processes requests sequentially. At 50 concurrent users, p99 latency hits ~25 seconds compared to vLLM's ~3 seconds. Do not use Ollama as a multi-user production server without load testing.**GGUF-only format.** Ollama only supports GGUF-quantized models. If you need FP16 inference, AWQ, or GPTQ formats, use vLLM or Transformers directly.**No built-in model quantization.** You cannot quantize a model within Ollama. Convert models to GGUF externally (using `llama.cpp/convert_hf_to_gguf.py` or similar), then import via `ollama create`.**Memory management is static.** `OLLAMA_MAX_LOADED_MODELS` controls how many models stay resident, but there is no dynamic VRAM balancing. On a 12 GB GPU, loading a 70B model (even Q4) will OOM — Ollama does not automatically offload layers to CPU.**Limited tool calling support.** While tool calling is available for compatible models (Llama 3.1+, Mistral), the implementatio```
bas
h
# Core settings
OLLAMA_HOST=0.0.0.0:11434          # Bind to all interfaces
OLLAMA_KEEP_ALIVE=24h               # Keep models loaded for 24 hours
OLLAMA_NUM_PARALLEL=4               # Max concurrent requests
OLLAMA_MAX_LOADED_MODELS=2          # Max models in VRAM simultaneously
OLLAMA_FLASH_ATTENTION=1            # Enable Flash Attention (faster inference)

# Performance tuning
OLLAMA_GPU_OVERHEAD=200MB           # Reserve VRAM headroom
OLLAMA_DEBUG=1                      # Verbose logging
```tomatical
l
y
. Performance depends on your CPU: an Intel i7-13700K generates ~8–12 tok/s with a 7B Q4 model. Apple Silicon M3 Pro achieves ~25 tok/s on CPU/Neural Engine.**Q: How do I update Ollama to the latest version?**
A: On macOS, run `brew upgrade ollama`. On Linux, re-run the install script: `curl -fsSL https://ollama.com/install.sh | sh`. The script preserves your downloaded models in `~/.ollama/models/`.**Q: Is Ollama suitable for production use?**
A: For single-purpose deployments (one model, one user, predictable```
ngin
x
server {
    listen 443 ssl http2;
    server_name ollama.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/ollama.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://localhost:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support for streaming
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        # Timeouts for long-running inference
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}
```vis
i
o
n
models for image analysis?**
A: Yes. Vision models like LLaVA 1.7, Qwen2-VL, and InternVL2.5 are supported. Pass image data as base64 in the chat API request. Note that vision models require significantly more VRAM (add ~2–4 GB overhead).### Self-Hosting NoteRunning this on your own VPS? Try DigitalOcean with $200 free credit
 — enough for 2 months of moderate self-hosting to test the setup risk-free. Best for low-medium traffic; scale to dedicated when you outgrow it.## ConclusionOllama removes the friction from local LLM deployment. One command installs it, one command pulls a model, and one command runs it. The Modelfile system gives you reproducible model customization. The OpenAI-compatible API means your existing LangChain, Open WebUI, and Continue.dev integrations work with a single URL change.For solo developers and small teams, Ollama is the pragmatic starting poi```
pytho
n
# ollama-auth-proxy.py (Flask example)
from flask import Flask, request, Response
import requests

app = Flask(__name__)
OLLAMA_URL = "http://localhost:11434"
VALID_KEYS = {"sk-your-api-key-here"}

@app.route('/', defaults={'path': ''}, methods=['GET', 'POST', 'PUT', 'DELETE'])
@app.route('/<path:path>', methods=['GET', 'POST', 'PUT', 'DELETE'])
def proxy(path):
    api_key = request.headers.get('Authorization', '').replace('Bearer ', '')
    if api_key not in VALID_KEYS:
        return {"error": "Invalid API key"}, 401
    
    resp = requests.request(
        method=request.method,
        url=f"{OLLAMA_URL}/{path}",
        headers={k: v for k, v in request.headers if k != 'Host'},
        data=request.get_data(),
        stream=True
    )
    return Response(resp.iter_content(chunk_size=1024), status=resp.status_code,
                   content_type=resp.headers.get('Content-Type'))

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=11435)
``` If you purchase services through these links, dibi8 may earn a commission at no additional cost to you.*







## Recommended Hosting & InfrastructureBefore you deploy any of the tools above into production, you'll need solid infrastructure. Two options dibi8 actually uses and recommends:- **DigitalOcean
** — $200 free credit for 60 days across 14+ global regions. The default option for indie devs running open-source AI tools.
- **HTStack
** — Hong Kong VPS with low-latency access from mainland China. This is the same IDC that hosts dibi8.com — battle-tested in production.*Affiliate links — they don't cost you extra and they help keep dibi8.com running.*## Sources & Further Reading- Ollama Official Documentation: https://docs.ollama.com
- Ollama GitHub Repository: https://github.com/ollama/ollama
- Ollama Model Library: https://ollama.com/search
- Ollama REST API Reference: https://docs.ollama.com/api
- Modelfile Reference: https://docs.```
bas
h
# List running models with memory usage
curl http://localhost:11434/api/ps

# Expected output:
# {
#   "models": [
#     {
#       "name": "llama3.2:8b",
#       "model": "llama3.2:8b",
#       "size": 5137025024,
#       "size_vram": 5137025024,
#       "expires_at": "2026-05-20T10:00:00Z"
#     }
#   ]
# }
```Oll
a
m
a
vs LocalAI: https://zenvanriel.com/ai-engineer-blog/ollama-vs-localai-comparison-local-model-deployment/
- Open WebUI GitHub: https://github.com/open-webui/open-webui
- LangChain Ollama Integration: https://python.langchain.com/docs/integrations/chat/ollama
- Continue.dev Documentation: https://docs.continue.dev<!--auto-references-->
## References & Sources- [Ollama](https://github.com/ollama/ollama)
- [Open WebUI](https://github.com/open-webui/open-webui)
- [llama.cpp](https://github.com/ggml-org```
in
i
# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama LLM Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=24h"

[Install]
WantedBy=default.target