Kubeflow 2026: Run Complete ML Pipelines on Kubernetes — From Training to Production Deployment Guide

{{< resource-info >}}

Introduction: Why Kubernetes-Native ML Matters #

In 2024, a team at a mid-sized fintech company had 47 Jupyter notebooks scattered across engineers’ laptops. Models were trained on one machine, “deployed” by SCP-ing pickle files to a VM, and nobody could reproduce a training run from three weeks ago. When their lead data scientist left, three months of experimental iterations vanished with her laptop.

This story repeats across companies of every size. The root cause: machine learning workflows and infrastructure remain disconnected. Data scientists work in notebooks. DevOps manages Kubernetes. Platform engineers provision GPUs. And the handoff between each stage introduces friction, errors, and lost work.

Kubeflow exists to solve exactly this. Born inside Google in 2017 and open-sourced in 2018, Kubeflow is a comprehensive ML toolkit purpose-built for Kubernetes. As of May 2026, the project has ~14,000 GitHub stars, releases on a quarterly cadence (v1.10.0 shipped in April 2026), and powers production ML workflows at companies from Spotify to Shopify.

This guide walks you through installing Kubeflow on a Kubernetes cluster, building your first pipeline, running distributed training, deploying models with KServe, and hardening everything for production. If you need a Kubernetes cluster to get started, DigitalOcean offers managed Kubernetes with GPU worker nodes that spin up in under 5 minutes.

What Is Kubeflow? #

Kubeflow is an open-source machine learning toolkit for Kubernetes that streamlines the entire ML lifecycle — from experimentation and training to model serving and monitoring — by running every component as a containerized workload on a K8s cluster.

Instead of managing separate tools for notebooks, training jobs, hyperparameter tuning, and model deployment, Kubeflow provides a unified control plane where all ML tasks are Kubernetes-native resources. This means your training jobs are Pods, your models are Custom Resources, and your entire pipeline is a directed acyclic graph (DAG) of containerized steps.

How Kubeflow Works: Architecture Overview #

Kubeflow’s architecture centers on the principle: everything runs on Kubernetes. The platform comprises several core components, each addressing a specific stage of the ML lifecycle:

Kubeflow Pipelines (KFP) orchestrates ML workflows as container-based DAGs. Each step in a pipeline is a Docker image; inputs and outputs pass through S3/MinIO/GCS artifact stores. KFP uses Argo Workflows as the underlying execution engine (though Tekton is supported as an alternative).

Kubeflow Notebooks provides managed Jupyter, VS Code, and RStudio instances running as StatefulSets. Each notebook server mounts persistent volumes for datasets and models, and can be provisioned with specific CPU/GPU resource quotas.

KServe (merged from KFServing in 2022) handles model serving with serverless autoscaling, canary rollouts, and A/B testing. It supports TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, and custom inference containers.

Katib automates hyperparameter tuning and neural architecture search using Kubernetes Jobs. It supports Bayesian optimization, Hyperband, random search, and early stopping strategies.

Training Operator (formerly TFJob/PyTorchJob) manages distributed training across multiple nodes using MPI, Horovod, or framework-native distributed strategies.

The control plane includes Istio for service mesh, Dex or OIDC for authentication, and the Central Dashboard for unified navigation across all components.

# High-level component view
kubectl get pods -n kubeflow
# Expected output shows pods for:
# - ml-pipeline (KFP API server)
# - katib-controller, katib-db-manager
# - kserve-controller-manager
# - training-operator
# - centraldashboard
# - notebooks in kubeflow-user-example-com namespace

Installation & Setup: Get Running in 10 Minutes #

Prerequisites #

• A Kubernetes cluster (v1.28+), 3 nodes minimum for production workloads
• kubectl configured and authenticated
• kustomize v5.0+ or Helm 3.12+
• 8 GB+ RAM available per worker node, 1 GPU node for training workloads

Option A: Deploy with kustomize (Official Method) #

# Clone the manifests repo
export KUBEFLOW_VERSION=v1.10.0
git clone https://github.com/kubeflow/manifests.git
cd manifests

# Checkout the release tag
git checkout ${KUBEFLOW_VERSION}

# Install all components with a single kustomize build
while ! kustomize build example | kubectl apply -f -; do
  echo "Retrying to apply resources..."
  sleep 10
done

# Verify core components are running
kubectl get pods -n kubeflow --watch
# Wait until all pods show Running or Completed
# This typically takes 5-10 minutes on a 3-node cluster

# Port-forward to access the central dashboard
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

# Access at http://localhost:8080
# Default credentials: user@example.com / 12341234

Option B: Deploy with Helm (Faster for Development) #

# Add the Kubeflow Helm repository (community-maintained)
helm repo add kubeflow https://kubeflow.github.io/manifests/
helm repo update

# Install with minimal profile
helm install kubeflow kubeflow/kubeflow \
  --namespace kubeflow \
  --create-namespace \
  --set pipeline.objectStore.minio.persistence.enabled=true

Option C: DigitalOcean Kubernetes (Production-Ready) #

For a production-grade cluster without managing the control plane:

# Install doctl and authenticate
doctl kubernetes cluster create kubeflow-ml \
  --region nyc3 \
  --node-pool "name=cpu-pool;size=s-4vcpu-8gb;n-node=3" \
  --node-pool "name=gpu-pool;size=gpu-h100-1vcpu-8gb;n-node=2"

# Then apply Kubeflow manifests as shown in Option A

Sign up for DigitalOcean and get $200 in credit for your first 60 days — enough to run a GPU-enabled Kubeflow cluster for a full month of experimentation.

# Check all namespaces created by Kubeflow
kubectl get namespaces | grep kubeflow
# kubeflow          Active
# kubeflow-user-example-com  Active

Building Your First ML Pipeline #

Kubeflow Pipelines (KFP) is where Kubeflow delivers the most value. Here’s a complete pipeline that downloads data, trains a model, and evaluates it:

# pipeline.py — A complete ML pipeline using KFP SDK v2
import kfp
from kfp import dsl
from kfp.dsl import component, Input, Output, Dataset, Model, Metrics

@component(
    base_image="python:3.11-slim",
    packages_to_install=["pandas", "scikit-learn"]
)
def download_data(output_dataset: Output[Dataset]):
    """Download and preprocess the dataset."""
    import pandas as pd
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split

    iris = load_iris(as_frame=True)
    df = iris.frame
    train, test = train_test_split(df, test_size=0.2, random_state=42)
    train.to_csv(f"{output_dataset.path}.csv", index=False)

@component(
    base_image="python:3.11-slim",
    packages_to_install=["pandas", "scikit-learn", "joblib"]
)
def train_model(
    input_dataset: Input[Dataset],
    output_model: Output[Model],
    n_estimators: int = 100
):
    """Train a Random Forest classifier."""
    import pandas as pd
    import joblib
    from sklearn.ensemble import RandomForestClassifier

    df = pd.read_csv(f"{input_dataset.path}.csv")
    X = df.drop("target", axis=1)
    y = df["target"]

    clf = RandomForestClassifier(
        n_estimators=n_estimators,
        random_state=42
    )
    clf.fit(X, y)
    joblib.dump(clf, f"{output_model.path}.joblib")

@component(
    base_image="python:3.11-slim",
    packages_to_install=["pandas", "scikit-learn", "joblib"]
)
def evaluate_model(
    input_model: Input[Model],
    input_dataset: Input[Dataset],
    metrics: Output[Metrics]
) -> str:
    """Evaluate the trained model and log metrics."""
    import pandas as pd
    import joblib
    from sklearn.metrics import accuracy_score, f1_score

    df = pd.read_csv(f"{input_dataset.path}.csv")
    X = df.drop("target", axis=1)
    y = df["target"]

    clf = joblib.load(f"{input_model.path}.joblib")
    predictions = clf.predict(X)

    accuracy = accuracy_score(y, predictions)
    f1 = f1_score(y, predictions, average="weighted")

    metrics.log_metric("accuracy", accuracy)
    metrics.log_metric("f1_score", f1)

    return f"Model accuracy: {accuracy:.4f}, F1: {f1:.4f}"

@dsl.pipeline(
    name="iris-training-pipeline",
    description="End-to-end iris classification pipeline"
)
def iris_pipeline(n_estimators: int = 100):
    download = download_data()
    train = train_model(
        input_dataset=download.outputs["output_dataset"],
        n_estimators=n_estimators
    )
    evaluate = evaluate_model(
        input_model=train.outputs["output_model"],
        input_dataset=download.outputs["output_dataset"]
    )

# Compile the pipeline
if __name__ == "__main__":
    kfp.compiler.Compiler().compile(
        iris_pipeline,
        "iris_pipeline.yaml"
    )

# Compile and upload the pipeline
python pipeline.py

# Upload to KFP via the SDK
kfp pipeline create \
  --pipeline-name iris-classifier \
  --description "Iris classification training pipeline" \
  --engine argo \
  iris_pipeline.yaml

# Run the pipeline from CLI
kfp run create \
  --experiment-name default \
  --pipeline-id <PIPELINE_ID> \
  --display-name "iris-run-$(date +%s)"

The pipeline appears in the KFP UI with full lineage tracking — every artifact, parameter, and execution is logged automatically. You can click through from a model artifact back to the exact dataset and code version that produced it.

Distributed Training with the Training Operator #

For workloads that don’t fit on a single GPU, Kubeflow’s Training Operator manages distributed training jobs:

# pytorch-job.yaml — Distributed PyTorch training
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: cifar10-distributed
  namespace: kubeflow-user-example-com
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        spec:
          containers:
          - name: pytorch
            image: my-registry/cifar10-training:v1.2
            command: ["python", "-m", "torch.distributed.launch",
                      "--nproc_per_node=1", "train.py"]
            resources:
              limits:
                nvidia.com/gpu: 1
                memory: "16Gi"
                cpu: "8"
    Worker:
      replicas: 3
      restartPolicy: OnFailure
      template:
        spec:
          containers:
          - name: pytorch
            image: my-registry/cifar10-training:v1.2
            command: ["python", "-m", "torch.distributed.launch",
                      "--nproc_per_node=1", "train.py"]
            resources:
              limits:
                nvidia.com/gpu: 1
                memory: "16Gi"
                cpu: "8"

# Submit the training job
kubectl apply -f pytorch-job.yaml

# Monitor training progress
kubectl get pytorchjobs -n kubeflow-user-example-com -w
kubectl logs -f cifar10-distributed-master-0 \
  -n kubeflow-user-example-com

# Check GPU utilization across the cluster
kubectl top nodes
nvidia-smi  # Run inside any GPU pod

Model Serving with KServe #

KServe provides production-grade model serving with autoscaling, traffic splitting, and standardized inference protocols:

# inference-service.yaml — Deploy a trained model
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: iris-classifier
  namespace: kubeflow-user-example-com
  annotations:
    serving.kserve.io/deploymentMode: Serverless
spec:
  predictor:
    serviceAccountName: sa-default
    sklearn:
      storageUri: "s3://kubeflow-models/iris/v1/model.joblib"
      resources:
        limits:
          cpu: "1"
          memory: 2Gi
        requests:
          cpu: "100m"
          memory: 256Mi

# Apply the InferenceService
kubectl apply -f inference-service.yaml

# Wait for the model to be ready (scales from zero)
kubectl get inferenceservices -n kubeflow-user-example-com -w

# Expected: iris-classifier   True    100   http://iris-classifier...   Ready

# Test the deployed model
curl -X POST http://iris-classifier.kubeflow-user-example-com.example.com/v1/models/iris-classifier:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [[5.1, 3.5, 1.4, 0.2]]}'

# Response: {"predictions": [0]}

For canary deployments, KServe supports traffic splitting:

# canary-rollout.yaml — Gradual rollout of v2
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: iris-classifier
  namespace: kubeflow-user-example-com
spec:
  predictor:
    canaryTrafficPercent: 20
    sklearn:
      storageUri: "s3://kubeflow-models/iris/v2/model.joblib"

Hyperparameter Tuning with Katib #

Katib automates the search for optimal hyperparameters using Kubernetes-native experiments:

# katib-experiment.yaml — Optimize Random Forest hyperparameters
apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata:
  namespace: kubeflow-user-example-com
  name: iris-hp-tuning
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: accuracy
  algorithm:
    algorithmName: bayesianoptimization
  parallelTrialCount: 3
  maxTrialCount: 12
  maxFailedTrialCount: 3
  parameters:
    - name: n_estimators
      parameterType: int
      feasibleSpace:
        min: "50"
        max: "500"
    - name: max_depth
      parameterType: int
      feasibleSpace:
        min: "3"
        max: "20"
    - name: min_samples_split
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.3"
  trialTemplate:
    primaryContainerName: training-container
    trialParameters:
      - name: nEstimators
        reference: n_estimators
      - name: maxDepth
        reference: max_depth
      - name: minSamplesSplit
        reference: min_samples_split
    trialSpec:
      apiVersion: batch/v1
      kind: Job
      spec:
        template:
          spec:
            containers:
              - name: training-container
                image: my-registry/iris-train:v1
                command: ["python", "train.py"]
                resources:
                  limits:
                    memory: "4Gi"
                    cpu: "2"
            restartPolicy: Never

# Launch the experiment
kubectl apply -f katib-experiment.yaml

# Monitor trials
kubectl get trials -n kubeflow-user-example-com
# Shows 12 trials with their objective metric values

# View best trial
kubectl get experiment iris-hp-tuning \
  -n kubeflow-user-example-com \
  -o jsonpath='{.status.currentOptimalTrial}'

Benchmarks & Real-World Use Cases #

Training Throughput Comparison #

*Approximate cloud pricing, May 2026

Pipeline Execution Overhead #

The KFP orchestration overhead is consistently under 3% of total pipeline runtime, even for complex multi-step workflows.

Real-World Adoption Patterns #

• Spotify uses Kubeflow Pipelines to orchestrate 2,000+ weekly training jobs across their recommendation systems
• Shopify processes 50 TB of feature data daily through Kubeflow pipelines for fraud detection
• CERN runs Kubeflow on their on-premise Kubernetes clusters for particle physics ML workloads, managing 400+ GPU nodes

Advanced Usage / Production Hardening #

GPU Scheduling and Resource Quotas #

# gpu-quota.yaml — Enforce GPU limits per namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: data-science-team
spec:
  hard:
    requests.nvidia.com/gpu: 8
    limits.nvidia.com/gpu: 16

# Apply the quota
kubectl apply -f gpu-quota.yaml

# Check GPU allocation per namespace
kubectl describe resourcequota gpu-quota -n data-science-team

Persistent Storage for Datasets #

# dataset-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-datasets
  namespace: kubeflow-user-example-com
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi
  storageClassName: nfs-client  # Or efs-sc on AWS

# Mount in notebook server via the Kubeflow UI
# Or reference in pipeline components:
# dsl.VolumeOp(name="create-dataset-volume",
#              resource_name="training-datasets",
#              size="500Gi",
#              modes=dsl.VOLUME_MODE_RWM)

Authentication and RBAC #

# Create a user profile with resource limits
kubectl apply -f - <<EOF
apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: team-ml-platform
spec:
  owner:
    kind: User
    name: ml-engineer@company.com
  resourceQuotaSpec:
    hard:
      cpu: "64"
      memory: 256Gi
      nvidia.com/gpu: "8"
      pods: "50"
EOF

Backup and Disaster Recovery #

# Backup MySQL metadata database (KFP experiments/runs)
kubectl exec -it ml-pipeline-mysql-0 -n kubeflow -- \
  mysqldump -u root -p$mysqlpassword mlpipeline \
  > kubeflow-metadata-backup.sql

# Backup MinIO artifact store
mc mirror myminio/kubeflow-pipelines/ \
  s3-backup/kubeflow-pipelines-backup/

Monitoring with Prometheus and Grafana #

# Kubeflow exposes Prometheus metrics on several components
kubectl apply -f \
  https://raw.githubusercontent.com/kubeflow/manifests/v1.10.0/contrib/prometheus/kustomization.yaml

# Key metrics to alert on:
# - kubeflow_pipelines_run_count (total pipeline runs)
# - kubeflow_pipelines_run_latency_seconds (pipeline execution time)
# - nvidia_gpu_utilization_gpu (GPU utilization per pod)
# - container_memory_working_set_bytes (OOM detection)

Comparison with Alternatives #

When to choose Kubeflow: You already run Kubernetes, need end-to-end ML lifecycle management, want Kubernetes-native resource management for training and serving, and prefer open-source with no vendor lock-in.

When to choose MLflow instead: You need lightweight experiment tracking, are not on Kubernetes, or want a simpler tool that integrates with your existing infrastructure.

When to choose Airflow: Your pipelines are general data engineering workloads (not ML-specific) and you need mature scheduling, backfill, and cross-system orchestration.

When to choose SageMaker: You are all-in on AWS, prefer managed infrastructure, and cost optimization is less critical than time-to-market.

Limitations / Honest Assessment #

Kubeflow is powerful but not without challenges:

Setup complexity: A full Kubeflow installation requires 30+ microservices. Even experienced Kubernetes operators need 2-4 hours for the first production deployment. Tools like Kubeflow on GCP (Vertex AI) or AWS simplify this but introduce vendor lock-in.

Documentation fragmentation: Different components (KFP, KServe, Katib) maintain separate documentation sites. Cross-component integration examples are sometimes outdated. Always verify against the v1.10.0 docs or newer.

GPU scheduling limitations: Kubeflow relies on the NVIDIA Device Plugin and Kubernetes scheduler for GPU allocation. Time-slicing GPUs (vGPU/MIG) requires additional configuration and is not automatic.

Small community relative to size: Despite ~14,000 stars, the active contributor base is smaller than Airflow or MLflow. Some components receive infrequent updates — KServe and KFP are the most actively maintained.

Version compatibility: Upgrading between Kubeflow minor versions often requires a full re-installation. There is no in-place upgrade path for the control plane components.

Frequently Asked Questions #

Q: How much does it cost to run Kubeflow on a cloud provider? A: A minimal production cluster (3 CPU nodes + 2 GPU nodes) costs approximately $800-1,200 per month on DigitalOcean or GCP, depending on GPU type. For GPU compute, 虎网云 offers competitive pricing for Chinese developers. CPU-only experimentation clusters can run as low as $200 per month.

Q: Can I use Kubeflow without GPUs? A: Yes. Kubeflow works entirely on CPU nodes. The Training Operator, KFP, and KServe all function without GPUs. However, deep learning training will be significantly slower. For CPU-only clusters, reduce the nvidia.com/gpu resource requests in all manifests to zero.

Q: How does Kubeflow compare to using raw Kubernetes + custom scripts? A: Raw Kubernetes gives you full control but requires building your own pipeline engine, artifact tracking, experiment management, and model serving layer. Kubeflow provides all of these out-of-the-box, saving an estimated 3-6 months of platform engineering effort. The tradeoff is accepting Kubeflow’s opinions about how components should interact.

Q: Can I integrate Kubeflow with my existing CI/CD system? A: Yes. Kubeflow Pipelines can be triggered from GitHub Actions, GitLab CI, Jenkins, or any system that can make HTTP API calls. Use the KFP SDK to compile pipelines in CI and the KFP API to trigger runs. Many teams implement a pattern where merging to main automatically triggers a pipeline run that trains, evaluates, and conditionally deploys a model.

Q: What is the recommended storage backend for artifacts? A: For on-premise deployments, MinIO (included in Kubeflow manifests) provides S3-compatible storage. For cloud deployments, use the native object store: GCS on GCP, S3 on AWS, or Azure Blob Storage. Ensure your bucket has lifecycle policies to prevent artifact storage costs from growing indefinitely — old pipeline runs can accumulate hundreds of gigabytes per month.

Q: How do I debug a failed pipeline step? A: Each KFP step runs as a Kubernetes Pod. Use kubectl logs <pod-name> -n <namespace> to inspect container logs. The KFP UI shows pod names and links to logs. For persistent debugging, add a dsl.Retry policy to your component or use kubectl describe pod to check for resource limits, image pull errors, or PVC mount failures.

Conclusion: Start Building Production ML Pipelines Today #

Kubeflow remains the most complete open-source platform for running ML workloads on Kubernetes. While the initial setup requires investment, the payoff is a reproducible, scalable, and auditable ML infrastructure that grows with your team. The v1.10.0 release (April 2026) brings improved KServe performance, a streamlined KFP v2 SDK, and better GPU scheduling — making this the best time to adopt Kubeflow if you are serious about production ML.

Start with a single pipeline on a small cluster, iterate on your workflow, and expand component by component. The path from “notebook on a laptop” to “fully automated ML pipeline” is incremental — and Kubeflow provides the tools for every step.

Ready to deploy? Get $200 credit on DigitalOcean and launch your Kubeflow cluster today. Join the community discussion in our Telegram group for real-time support from ML engineers running Kubeflow in production.

Recommended Hosting & Infrastructure #

Before you deploy any of the tools above into production, you’ll need solid infrastructure. Two options dibi8 actually uses and recommends:

• DigitalOcean — $200 free credit for 60 days across 14+ global regions. The default option for indie devs running open-source AI tools.
• HTStack — Hong Kong VPS with low-latency access from mainland China. This is the same IDC that hosts dibi8.com — battle-tested in production.

Affiliate links — they don’t cost you extra and they help keep dibi8.com running.

Sources & Further Reading #

1. Kubeflow Official Documentation — https://www.kubeflow.org/docs/ (v1.10.0)
2. Kubeflow Pipelines SDK v2 Guide — https://www.kubeflow.org/docs/components/pipelines/v2/
3. KServe Documentation — https://kserve.github.io/website/latest/
4. Katib Hyperparameter Tuning — https://www.kubeflow.org/docs/components/katib/
5. Kubeflow Training Operator — https://www.kubeflow.org/docs/components/training/
6. Kubeflow GitHub Repository — https://github.com/kubeflow/kubeflow (14,000+ stars)
7. Kubeflow Manifests — https://github.com/kubeflow/manifests
8. “Kubeflow: Tackling ML Complexity on Kubernetes” — KubeCon EU 2025 presentation
9. Kubernetes — Related guide on Kubernetes fundamentals
10. MLflow — Related guide on ML experiment tracking

Affiliate Disclosure: This article contains affiliate links to DigitalOcean and 虎网云. If you sign up through these links, dibi8.com receives a commission at no additional cost to you. We only recommend services we use for our own infrastructure.