VideoReTalking: 7.2K+ Stars — AI Lip-Sync Video Editing Setup Guide 2026

{{< resource-info >}}

Introduction #

Dubbing a video in another language while keeping the lip movements synchronized has been a post-production nightmare for years. Manual frame-by-frame adjustments take hours per minute of footage, and the results rarely look natural. In 2022, researchers from Xidian University and Tencent AI Lab published VideoReTalking at SIGGRAPH Asia — a system that automates this by editing the lower face region of existing talking-head videos to match any input audio. With 7,200+ GitHub stars and a fully open-source Apache-2.0 license, it remains one of the most practical self-hosted lip-sync solutions available in 2026. This guide walks through a complete VideoReTalking tutorial: installation, inference, Gradio UI setup, TTS integration, and production deployment.

What Is VideoReTalking? #

VideoReTalking is a PyTorch-based inference pipeline that takes a talking-head video and a target audio clip as inputs, then produces a lip-synced output video where the subject’s mouth movements match the new audio. Unlike text-to-speech avatars that generate video from scratch, VideoReTalking works with existing footage — preserving the original lighting, background, head pose, and visual quality while only modifying the lip region.

How VideoReTalking Works #

VideoReTalking uses a three-stage architecture that disentangles expression, lip-sync, and enhancement into separate modules:

Stage 1: D-Net — Expression Normalization #

The D-Net (Expression Editing Network) takes the input video and standardizes the facial expression across all frames to a neutral template. It extracts 3DMM coefficients from each frame using DECA-based face reconstruction, replaces the expression parameters with a predefined neutral template, and synthesizes a stabilized video. This step prevents the lip-sync network from being influenced by the original mouth movements.

Stage 2: L-Net — Audio-Driven Lip Synchronization #

The L-Net (Lip-Sync Network) receives the expression-normalized video and the target audio. It uses a Wav2Lip-based generator with audio-visual feature fusion to produce lip movements that match the input audio. The network outputs a video with synchronized lips but potentially lower visual fidelity around the mouth region.

Stage 3: E-Net — Face Enhancement #

The E-Net (Enhancement Network) uses GFPGAN and GPEN face restoration models to improve photo-realism. It performs identity-aware face enhancement, blends the enhanced face back onto the original frame using Laplacian pyramid blending, and preserves the subject’s identity characteristics throughout the pipeline.

Installation & Setup #

VideoReTalking pipeline overview — editing talking-head videos to match any target audio while preserving original visual quality.

Hardware Requirements #

CPU-only inference is supported but runs 10–15x slower. Apple Silicon (M1/M2) works in CPU mode.

Step 1: Clone the Repository #

git clone https://github.com/OpenTalker/video-retalking.git
cd video-retalking

Step 2: Create Conda Environment #

conda create -n video_retalking python=3.8 -y
conda activate video_retalking
conda install ffmpeg -y

Step 3: Install PyTorch with CUDA #

# For CUDA 11.1 (original project default)
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

# For CUDA 12.1 (modern GPUs, 2026)
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121

Step 4: Install Dependencies #

pip install -r requirements.txt

The requirements.txt installs the following key packages:

basicsr==1.4.2
kornia==0.5.1
face-alignment==1.3.4
ninja==1.10.2.3
einops==0.4.1
facexlib==0.2.5
librosa==0.9.2
dlib==19.24.0
gradio>=3.7.0
numpy==1.23.4

Step 5: Download Pre-trained Models #

Download the pre-trained checkpoints from Google Drive and extract them into ./checkpoints/:

# Directory structure should look like:
# ./checkpoints/
#   ├── 244000.pth          (D-Net expression editing)
#   ├── wav2lip.pth         (L-Net lip sync)
#   ├── GFPGANv1.3.pth      (GFPGAN enhancer)
#   ├── GPEN-BFR-512.pth    (GPEN enhancer)
#   └── ...

Step 6: Verify Installation #

python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU')"

Expected output on a GPU system:

CUDA available: True
Device: NVIDIA GeForce RTX 4090

Integration with TTS and Voice Cloning Tools #

Integration with RVC (Retrieval-based Voice Conversion) #

RVC converts one voice to another while preserving prosody. Chain it with VideoReTalking for voice-swapped lip-synced output:

# Step 1: Generate or convert audio with RVC
python rvc/infer.py --input input.wav --model weights/model.pth --output rvc_output.wav

# Step 2: Feed RVC output into VideoReTalking
python inference.py \
  --face input_video.mp4 \
  --audio rvc_output.wav \
  --outfile output_rvc_synced.mp4

Integration with GPT-SoVITS #

GPT-SoVITS generates high-quality TTS with few-shot voice cloning. The workflow:

# gpt_sovits_videoretalking.py
import subprocess
import os

# Step 1: Generate TTS with GPT-SoVITS
subprocess.run([
    "python", "GPT_SoVITS/inference_webui.py",
    "--text", "Hello, this is the dubbed version of my video.",
    "--ref_audio", "reference_voice.wav",
    "--output", "tts_output.wav"
])

# Step 2: Run VideoReTalking
subprocess.run([
    "python", "inference.py",
    "--face", "source_video.mp4",
    "--audio", "tts_output.wav",
    "--outfile", "final_dubbed.mp4",
    "--exp_img", "neutral",
    "--up_face", "surprise"
])

Integration with Coqui TTS #

# Install Coqui TTS
pip install TTS

# Generate speech
tts --text "This is the new dialogue for the video." \
    --model_name tts_models/multilingual/multi-dataset/xtts_v2 \
    --speaker_wav reference.wav \
    --language_idx en \
    --out_path coqui_output.wav

# Lip-sync with VideoReTalking
python inference.py \
  --face original_video.mp4 \
  --audio coqui_output.wav \
  --outfile coqui_synced.mp4

Benchmarks / Real-World Use Cases #

Inference Speed Benchmarks #

Tested on an NVIDIA RTX 4090 with a 10-second 512x512 input video:

VideoReTalking processes roughly 1 second of video per 1 second of GPU time at 512x512 resolution on a modern GPU.

Video Quality Benchmarks #

Higher LSE-C = better audio-lip alignment. Lower LPIPS = closer to ground truth.

Real-World Use Cases #

Example input frame and processed output showing the lip-sync quality VideoReTalking produces.

1. Video Dubbing: Localize training videos, marketing content, and e-learning materials by dubbing in multiple languages while preserving the original speaker’s visual presence.
2. Podcast-to-Video Enhancement: Match pre-recorded audio commentary with talking-head footage when the original recording had audio issues.
3. Virtual Avatars: Combine with TTS engines to create real-time avatar systems for customer service or streaming applications.
4. Content Restoration: Fix out-of-sync footage caused by encoding errors or multi-camera editing issues.

Advanced Usage / Production Hardening #

Gradio WebUI Setup #

VideoReTalking includes a built-in Gradio interface for browser-based usage:

# Launch the WebUI
python webUI.py

The WebUI will start on http://localhost:7860 by default. It supports:

• Drag-and-drop video and audio upload
• Expression template selection (neutral, smile)
• Upper-face emotion control (surprise, angry)
• Batch segment processing for long videos

For remote access behind a reverse proxy:

python webUI.py --server-name 0.0.0.0 --server-port 7860 --share

Docker Deployment #

# Dockerfile
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y \
    python3.8 python3-pip ffmpeg git wget \
    libgl1-mesa-glx libglib2.0-0

WORKDIR /app
RUN git clone https://github.com/OpenTalker/video-retalking.git .
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
RUN pip3 install -r requirements.txt

# Download checkpoints (mount as volume in production)
RUN mkdir -p checkpoints

EXPOSE 7860
CMD ["python3", "webUI.py", "--server-name", "0.0.0.0"]

Build and run:

docker build -t video-retalking .
docker run --gpus all -p 7860:7860 -v $(pwd)/checkpoints:/app/checkpoints video-retalking

Batch Processing Script #

#!/usr/bin/env python3
# batch_process.py
import os
import subprocess
from pathlib import Path

INPUT_DIR = "input_videos"
AUDIO_DIR = "input_audio"
OUTPUT_DIR = "output"
os.makedirs(OUTPUT_DIR, exist_ok=True)

video_files = sorted(Path(INPUT_DIR).glob("*.mp4"))
audio_files = sorted(Path(AUDIO_DIR).glob("*.wav"))

for vid, aud in zip(video_files, audio_files):
    outname = f"{OUTPUT_DIR}/{vid.stem}_synced.mp4"
    print(f"Processing: {vid.name} + {aud.name}")
    subprocess.run([
        "python", "inference.py",
        "--face", str(vid),
        "--audio", str(aud),
        "--outfile", outname,
        "--exp_img", "neutral"
    ])

Monitoring and Logging #

# Production wrapper with structured logging
import logging
import time
import torch

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('videoretalking.log'),
        logging.StreamHandler()
    ]
)

def inference_with_monitoring(face_path, audio_path, output_path):
    start = time.time()
    vram_before = torch.cuda.memory_allocated() / 1e9
    
    # Run inference
    subprocess.run([...])  # inference command
    
    elapsed = time.time() - start
    vram_after = torch.cuda.memory_allocated() / 1e9
    
    logging.info(f"Processed {face_path} in {elapsed:.1f}s, "
                 f"VRAM: {vram_before:.1f}GB -> {vram_after:.1f}GB")

Security Considerations #

• Run inside a container with read-only filesystem mounts for model weights
• Restrict GPU access with CUDA_VISIBLE_DEVICES to isolate workloads
• Validate input file formats before processing to prevent path traversal
• The project includes a comprehensive disclaimer about portrait rights and compliance

Comparison with Alternatives #

VideoReTalking sits in the sweet spot between speed and quality. Wav2Lip is faster but produces blurrier output. SadTalker generates compelling head motion from a single image but requires a still photo input rather than video. GeneFace produces high fidelity but demands more VRAM and lacks CPU support.

Limitations / Honest Assessment #

VideoReTalking is not the right tool for every scenario:

1. Extreme head poses fail: The D-Net cannot handle extreme profile views or heavily occluded faces. Side-view videos beyond ±45° yaw will produce artifacts.
2. No real-time capability: The three-stage pipeline requires processing the entire video sequentially. Expect ~1x real-time inference at best — not suitable for live streaming without pre-buffering.
3. Resolution ceiling: The enhancement networks are trained on 512x512 face crops. Upscaling beyond this produces diminishing returns.
4. Expression consistency: While the expression templates work well, subtle micro-expressions from the original video are lost during D-Net normalization.
5. Audio quality dependency: Background music or noisy audio confuses the lip-sync network. Clean speech WAV files produce the best results.
6. Model download friction: The 2GB+ checkpoint download from Google Drive is not scriptable for CI/CD pipelines.

Frequently Asked Questions #

What hardware do I need to run VideoReTalking? #

An NVIDIA GPU with at least 8GB VRAM is the practical minimum. A 4090 with 24GB handles 512x512 video at roughly real-time speed. CPU inference works but takes 10–15x longer. Apple Silicon runs in CPU mode only.

Can I use VideoReTalking for commercial projects? #

Yes. The Apache-2.0 license permits commercial use. However, the project includes a disclaimer about portrait rights — you must obtain consent from anyone whose likeness is modified. The Tencent trademark may not be used without written permission.

How does VideoReTalking compare to cloud lip-sync APIs? #

Self-hosted VideoReTalking costs $0.05–0.20 per minute in electricity/GPU rental. Cloud APIs like Sync Labs or HeyGen charge $0.50–3.00 per minute but require no setup. VideoReTalking wins on privacy (fully offline) and cost at scale; cloud APIs win on ease of integration.

Does it support languages other than English? #

Yes. The lip-sync network is language-agnostic — it maps acoustic features to visemes (mouth shapes), which are shared across languages. Chinese, Japanese, Spanish, and Arabic have all been tested by the community with good results.

Why is my output video blurry around the mouth? #

The default GFPGAN enhancer applies a moderate smoothing effect. Try switching to the GPEN enhancer by editing inference.py and changing the enhancer initialization. Alternatively, disable enhancement entirely for sharper (but potentially less consistent) output.

Can I fine-tune the models on my own dataset? #

The repository provides inference code only. Fine-tuning would require reimplementing the training loops for D-Net (expression editing on VoxCeleb) and L-Net (lip-sync on LRS2). The paper includes sufficient architectural detail for this, but no training scripts are included.

Conclusion #

VideoReTalking offers a practical, self-hosted solution for audio-driven lip synchronization with production-grade output quality. The three-stage pipeline — expression normalization, lip-sync generation, and face enhancement — produces results that rival commercial alternatives at a fraction of the cost for high-volume workflows.

Action items to get started:

1. Clone the repo and set up the conda environment using the commands above
2. Download the 2GB checkpoint bundle into ./checkpoints/
3. Run the quick inference command with the sample files in examples/
4. Launch the Gradio WebUI for interactive experimentation
5. Chain with GPT-SoVITS or RVC for a complete voice-cloning + lip-sync pipeline

Join the dibi8 developer community on Telegram to share your VideoReTalking deployments and get help from other builders.

Recommended Hosting & Infrastructure #

Before you deploy any of the tools above into production, you’ll need solid infrastructure. Two options dibi8 actually uses and recommends:

• DigitalOcean — $200 free credit for 60 days across 14+ global regions. The default option for indie devs running open-source AI tools.
• HTStack — Hong Kong VPS with low-latency access from mainland China. This is the same IDC that hosts dibi8.com — battle-tested in production.

Affiliate links — they don’t cost you extra and they help keep dibi8.com running.

Sources & Further Reading #

• VideoReTalking GitHub Repository
• VideoReTalking Project Page
• VideoReTalking arXiv Paper
• SIGGRAPH Asia 2022 Proceedings
• Wav2Lip Repository
• SadTalker Repository
• GeneFace Repository
• GFPGAN Face Restoration
• Pre-trained Models (Google Drive)