字节开源 Bernini：统一视频生成与编辑框架，MLLM 语义规划 + DiT 渲染，比肩顶级商业模型

ByteDance Open-Sources Bernini: Unified Video Generation & Editing with MLLM Semantic Planning + DiT Rendering

Tech-Experiment #视频生成#字节跳动#开源模型#Bernini#Wan2.2#视频编辑#AI视频#MLLM#DiT#部署指南
更新于
🇨🇳 中文

GitHub：bytedance/Bernini
HuggingFace Collection：ByteDance/bernini
项目主页：bernini-ai.github.io
论文：arXiv 2605.22344
开源时间：2026年6月（Bernini-R），7月21日（训练代码完整开放）
许可证：Apache 2.0

这是什么

字节跳动 Bernini Team 开源的统一视频生成与编辑框架，Stars 已超 1100（今日刚完整开放训练代码）。

Bernini 的核心思路与众不同：不是直接让扩散模型猜怎么改，而是先让 MLLM 规划”要在语义空间做什么变化”，再让扩散渲染器执行。这条两阶段路线让它在复杂指令遵循上明显优于纯渲染器方案。

系统由两个组件组成：

MLLM 语义规划器（Semantic Planner）

  • 基础：Qwen2.5-VL-7B-Instruct
  • 接收：文本指令 + 源图像 / 源视频 + 参考图像
  • 输出：目标语义嵌入序列（在潜在空间预测出”要生成什么”）

DiT 渲染器（Renderer）

  • 基础：Wan2.2-T2V-A14B（MoE 架构，参数 14B）
  • 接收：语义嵌入 + VAE 潜变量
  • 执行：流匹配去噪，输出最终视频帧

两者通过 Segment-Aware 3D RoPE（SA-3D RoPE） 连接——这个改进的位置编码区分了来自不同视觉段（源视频帧、参考图像、目标位置）的 token，解决了多源输入时的对齐问题。

性能水平

官方 Human Arena 评测（人工盲测配对，Bradley-Terry 评分）：

排名方法BT 分数胜率
1HappyHorse-1.0（闭源商业）108061.3%
2Bernini（开源）104456.3%
3Wan2.7103454.9%
4Grok-imagine-video96444.9%

Bernini 是唯一进入该榜单前三的开源模型，与排名第一的闭源商业产品差距仅 36 分（约 5%）。

基准评测数据：

模型EditVerseOpenVEVBench
Bernini-R 1.3B7.743.6584.69
Bernini-R 14B7.993.7884.64
Bernini 7B+14B8.024.0384.37

两个可部署产品线

Bernini（完整流水线）

适合场景：复杂指令、多步语义规划、强调指令遵循精度
权重：ByteDance/Bernini-Diffusers（7B Planner + 14B Renderer，打包格式）
显存需求：推荐 8×H100/A100（80GB），也支持 4×A100 配合 offload

完整包目录结构：

ByteDance/Bernini-Diffusers/
  bernini/           ← Bernini 规划权重
  mllm/              ← Qwen2.5-VL-7B 规划器
  t5_text_encoder/   ← 文本编码器
  t5_tokenizer/
  vae/
  scheduler/
  transformer_config.json
  transformer_2_config.json

Bernini-R（仅渲染器）

适合场景：简单编辑（风格迁移、字幕/水印去除、局部修改）、更快推理、ComfyUI 集成
权重：ByteDance/Bernini-R-Diffusers（14B）或 ByteDance/Bernini-R-1.3B-Diffusers
显存需求：14B 需 8×GPU；1.3B 可单卡 24GB（社区已验证 RTX 4090）

环境要求

Python 3.11.2
CUDA 12.6（最低 12.3）
PyTorch 2.7.1+cu126
diffusers 0.35.2
accelerate 0.34.2
transformers 4.57.3

注意事项：

  • H100/H800/H200（Hopper）：可启用 FlashAttention-3，推理最快
  • A100/A800：使用 FlashAttention-2，性能良好
  • 其他 CUDA GPU：回退到 PyTorch SDPA
  • CPU / Apple Silicon：官方暂不支持（需 CUDA）

完整部署步骤

第一步：安装依赖

git clone https://github.com/bytedance/Bernini.git bernini
cd bernini
pip install -r requirements.txt

# 多 GPU 序列并行必须安装 VeOmni（--no-deps 避免覆盖 torch 版本）
pip install --no-deps git+https://github.com/ByteDance-Seed/VeOmni.git@v0.1.11

# 可选：FlashAttention-2（A100 及以下）
pip install flash-attn==2.8.3

# 可选：FlashAttention-3（H100 专属，需从源码编译）
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention && git checkout v2.8.3
cd hopper && MAX_JOBS=$(nproc) python3 setup.py install --user

第二步：下载权重

选 Bernini-R（推荐入门）：

pip install -U "huggingface_hub"

# 14B 完整版（~28GB）
hf download ByteDance/Bernini-R-Diffusers \
    --local-dir pretrained_models/Bernini-R-Diffusers

# 或 1.3B 轻量版（~3GB，适合 24GB 单卡）
hf download ByteDance/Bernini-R-1.3B-Diffusers \
    --local-dir pretrained_models/Bernini-R-1.3B-Diffusers

选 Bernini 完整流水线：

hf download ByteDance/Bernini-Diffusers \
    --local-dir pretrained_models/Bernini-Diffusers

国内网络建议走 ModelScope 镜像（可配 HF_ENDPOINT=https://hf-mirror.com）。

第三步：了解 Case File 格式

Bernini 用 JSON Case File 传递任务参数，而不是长命令行标志：

{
  "task_type": "v2v",
  "guidance_mode": "v2v_apg",
  "prompt": "Remove the white sheep on the left side of the video.",
  "video": "path/to/source.mp4",
  "output": "output/edited.mp4"
}

任务类型（task_type）：

  • t2i：文本→图像
  • i2i：图像编辑
  • t2v：文本→视频
  • v2v：视频编辑
  • rv2v：参考图像引导视频编辑
  • r2v：参考图像→视频生成

6 大任务类型实战命令

1. 文本生成图像（t2i）— 单卡

python infer_single_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --case assets/testcases/t2i/t2i.json \
    --num_frames 1 \
    --guidance_mode t2v_apg

或直接传参数：

python infer_single_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --prompt "A futuristic cityscape at dusk, cinematic lighting, 8K" \
    --task_type t2i \
    --num_frames 1 \
    --output output/city.png

2. 图像编辑（i2i）— 单卡

python infer_single_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --case assets/testcases/i2i/i2i.json \
    --num_frames 1 \
    --guidance_mode t2v_apg

3. 文本生成视频（t2v）— 多卡

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/t2v/t2v.json \
    --guidance_mode t2v_apg

默认输出：480p / 16fps / 81帧（约5秒）

4. 视频编辑（v2v）— 多卡

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/v2v/v2v_case1.json \
    --guidance_mode v2v_apg

Case 文件示例（天气改变）：

{
  "task_type": "v2v",
  "guidance_mode": "v2v_apg",
  "prompt": "Convert the video into an immersive snowy winter wonderland.",
  "video": "assets/source_videos/forest.mp4",
  "output": "output/winter.mp4"
}

5. 参考图像引导编辑（rv2v）— 多卡

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/rv2v/rv2v_case1.json \
    --guidance_mode rv2v_apg

适合：用参考图替换视频中的物体、材质、天气、风格。

6. 参考→视频生成（r2v）— 最多5张参考图

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/r2v/r2v_case1.json \
    --guidance_mode r2v_apg

使用脚本批量运行：

# 一键运行各任务（读取 BERNINI_R_CONFIG 环境变量）
export BERNINI_R_CONFIG=./pretrained_models/Bernini-R-Diffusers
export NPROC_PER_NODE=8
export ULYSSES=8

bash scripts/bernini_r/run_t2i.sh
bash scripts/bernini_r/run_t2v.sh
bash scripts/bernini_r/run_v2v.sh
bash scripts/bernini_r/run_rv2v.sh

Gradio 可视化界面

# 单卡（仅图像任务）
python gradio_demo.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --port 7860

# 8 卡并行（视频任务）
torchrun --nproc-per-node 8 gradio_demo.py \
    --ulysses 8 \
    --config pretrained_models/Bernini-R-Diffusers \
    --port 7860 \
    --share   # 生成公开 URL

Prompt Enhancer（强烈推荐）

启用 --use_pe 可通过任意 OpenAI 兼容端点增强提示词，显著提升生成质量。

export BERNINI_PE_API_KEY=your_key
export BERNINI_PE_BASE_URL=https://api.openai.com/v1   # 或 Ollama/vLLM 端点
export BERNINI_PE_MODEL=gpt-4o-mini                     # 任意视觉模型

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/t2v/t2v.json \
    --use_pe

配合本地模型（免费，完全离线）：

# 启动 Ollama
ollama serve &
ollama pull qwen2.5vl:7b

export BERNINI_PE_API_KEY=ollama
export BERNINI_PE_BASE_URL=http://localhost:11434/v1
export BERNINI_PE_MODEL=qwen2.5vl:7b

完整流水线（Bernini 7B+14B）特有命令

export BERNINI_CONFIG=./pretrained_models/Bernini-Diffusers
export NPROC_PER_NODE=8
export ULYSSES=8

# 文本生成视频（更强的指令跟随）
bash scripts/bernini/run_t2v.sh

# 复杂视频编辑（MLLM 语义规划优势最明显的场景）
CASE_PATH=assets/testcases/v2v/v2v_case2.json \
bash scripts/bernini/run_v2v.sh

# Gradio 界面
torchrun --nproc-per-node 8 gradio_demo.py \
    --ulysses 8 \
    --config ByteDance/Bernini-Diffusers \
    --port 7860 --share

显存配置参考

模型GPU 配置分辨率备注
Bernini-R 1.3B单卡 RTX 4090 (24GB)480p社区验证可行
Bernini-R 14B8×A100 (80GB)480p/720p官方推荐
Bernini-R 14B4×A100 (80GB)480p减少 --ulysses 4
Bernini-R 14B8×H100 (80GB)480p/720p最优，FlashAttn-3
Bernini 7B+14B8×H100 (80GB)480p/720p完整流水线推荐

国内可用的 A100/H100 算力租用：AutoDL、Vast.ai、Lepton.ai（按需选择）

训练（Fine-tune Bernini-R）

训练代码于 2026-07-13 完整开放：

# 推荐用 uv 管理训练环境
curl -LsSf https://astral.sh/uv/install.sh | sh

uv sync
uv sync --extra all
uv pip install --no-build-isolation "flash-attn==2.8.3"

# 开始训练
uv run python train_bernini_r.py \
    --config configs/bernini_renderer_wan22/config.json \
    --dataset_path /path/to/your/dataset \
    --output_dir output/finetuned

完整训练文档：docs/bernini_r_train.md

关键技术设计

为什么不用向量数据库式的”检索图像”而是”规划语义”？

传统视频编辑方法直接把源视频和提示词丢给扩散模型，依赖模型自己”猜”出正确的编辑方向。对于简单的风格迁移这没问题，但复杂指令（“把视频右半部分变成俄罗斯古典宫廷舞蹈黏土动画停格风格，左半保持原战争场景”）很难一步到位。

Bernini 的解法是：先让 MLLM 规划”目标语义嵌入在哪里”，再让 DiT 沿着语义梯度去噪。这把一个硬推理问题拆成了两个更简单的子问题。

SA-3D RoPE 解决了什么？

多源输入（源视频帧 + 参考图像1 + 参考图像2 + 目标占位符）会在 attention 中混乱。SA-3D RoPE 为不同视觉段的 token 赋予不同的位置编码，让渲染器知道哪些 token 来自源，哪些是参考，哪些是要生成的目标。

与同类开源模型对比

模型机构规划器渲染器最强任务单卡可运行
Bernini字节跳动Qwen2.5-VL-7BWan2.2-14B复杂视频编辑仅 1.3B 版本
Wan2.2阿里无MoE-14BT2V 生成5B 版 720P
HunyuanVideo腾讯无13BT2V 生成部分支持
CogVideoX智谱无5B/13BT2V 生成5B 可单卡

Bernini 目前是开源生态里视频编辑（v2v）能力最强的，但 T2V 纯生成能力不是它的重点（VBench 84.37 对比 Wan2.2 的 top 性能略低）。

上手建议

  1. 从 Bernini-R 1.3B 开始：单卡 24GB 可跑，先验证 i2i（图像编辑）任务流通了
  2. 用 Gradio 界面测试：--share 生成公开 URL，不需要本地 UI
  3. v2v 是杀手用例：天气变换、风格迁移、对象删除是 Bernini 最闪亮的场景
  4. 接 Prompt Enhancer：配本地 Ollama（Qwen2.5-VL），提示词质量差距非常大
  5. 复杂指令上完整 Bernini 7B+14B：1.3B 在”人物动作生成”等复杂任务上明显弱于 14B

参考资源

© 2026 Author: Mycelium Protocol

🇬🇧 English

GitHub: bytedance/Bernini HuggingFace Collection: ByteDance/bernini Project Homepage: bernini-ai.github.io Paper: arXiv 2605.22344 Open-Source Date: June 2026 (Bernini-R), July 21 (full training code released) License: Apache 2.0

What Is This

An open-source unified video generation and editing framework from ByteDance’s Bernini Team, which has already surpassed 1,100 Stars (training code fully opened today).

Bernini’s core approach is distinctive: rather than asking a diffusion model to guess how to make edits directly, it first has an MLLM plan “what changes to make in semantic space,” then has a diffusion renderer execute them. This two-stage pipeline gives it a clear edge over pure-renderer approaches when it comes to following complex instructions.

The system consists of two components:

MLLM Semantic Planner

  • Base: Qwen2.5-VL-7B-Instruct
  • Input: text instructions + source image / source video + reference images
  • Output: target semantic embedding sequences (predicting “what to generate” in latent space)

DiT Renderer

  • Base: Wan2.2-T2V-A14B (MoE architecture, 14B parameters)
  • Input: semantic embeddings + VAE latent variables
  • Execution: flow-matching denoising to produce final video frames

The two are connected through Segment-Aware 3D RoPE (SA-3D RoPE) — an improved positional encoding that distinguishes tokens from different visual segments (source video frames, reference images, target positions), resolving the alignment problem that arises with multi-source inputs.

Performance

Official Human Arena evaluation (human blind pairwise testing, Bradley-Terry scoring):

RankMethodBT ScoreWin Rate
1HappyHorse-1.0 (closed-source commercial)108061.3%
2Bernini (open-source)104456.3%
3Wan2.7103454.9%
4Grok-imagine-video96444.9%

Bernini is the only open-source model to reach the top three on this leaderboard, trailing the top-ranked closed-source commercial product by only 36 points (approximately 5%).

Benchmark evaluation data:

ModelEditVerseOpenVEVBench
Bernini-R 1.3B7.743.6584.69
Bernini-R 14B7.993.7884.64
Bernini 7B+14B8.024.0384.37

Two Deployable Product Lines

Bernini (Full Pipeline)

Best for: complex instructions, multi-step semantic planning, emphasis on instruction-following precision Weights: ByteDance/Bernini-Diffusers (7B Planner + 14B Renderer, packaged format) VRAM requirement: Recommended 8×H100/A100 (80GB); also supports 4×A100 with offloading

Full package directory structure:

ByteDance/Bernini-Diffusers/
  bernini/           ← Bernini planning weights
  mllm/              ← Qwen2.5-VL-7B planner
  t5_text_encoder/   ← text encoder
  t5_tokenizer/
  vae/
  scheduler/
  transformer_config.json
  transformer_2_config.json

Bernini-R (Renderer Only)

Best for: simple edits (style transfer, subtitle/watermark removal, local modifications), faster inference, ComfyUI integration Weights: ByteDance/Bernini-R-Diffusers (14B) or ByteDance/Bernini-R-1.3B-Diffusers VRAM requirement: 14B requires 8×GPU; 1.3B can run on a single 24GB card (community-verified on RTX 4090)

System Requirements

Python 3.11.2
CUDA 12.6 (minimum 12.3)
PyTorch 2.7.1+cu126
diffusers 0.35.2
accelerate 0.34.2
transformers 4.57.3

Notes:

  • H100/H800/H200 (Hopper): can enable FlashAttention-3 for fastest inference
  • A100/A800: uses FlashAttention-2, good performance
  • Other CUDA GPUs: falls back to PyTorch SDPA
  • CPU / Apple Silicon: not officially supported (requires CUDA)

Full Deployment Steps

Step 1: Install Dependencies

git clone https://github.com/bytedance/Bernini.git bernini
cd bernini
pip install -r requirements.txt

# Multi-GPU sequence parallelism requires VeOmni (--no-deps to avoid overwriting torch version)
pip install --no-deps git+https://github.com/ByteDance-Seed/VeOmni.git@v0.1.11

# Optional: FlashAttention-2 (A100 and below)
pip install flash-attn==2.8.3

# Optional: FlashAttention-3 (H100 exclusive, must compile from source)
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention && git checkout v2.8.3
cd hopper && MAX_JOBS=$(nproc) python3 setup.py install --user

Step 2: Download Weights

Choose Bernini-R (recommended for beginners):

pip install -U "huggingface_hub"

# 14B full version (~28GB)
hf download ByteDance/Bernini-R-Diffusers \
    --local-dir pretrained_models/Bernini-R-Diffusers

# Or 1.3B lightweight version (~3GB, suitable for single 24GB card)
hf download ByteDance/Bernini-R-1.3B-Diffusers \
    --local-dir pretrained_models/Bernini-R-1.3B-Diffusers

Choose Bernini full pipeline:

hf download ByteDance/Bernini-Diffusers \
    --local-dir pretrained_models/Bernini-Diffusers

For users in mainland China, ModelScope mirror is recommended (set HF_ENDPOINT=https://hf-mirror.com).

Step 3: Understand the Case File Format

Bernini uses JSON Case Files to pass task parameters, rather than long command-line flags:

{
  "task_type": "v2v",
  "guidance_mode": "v2v_apg",
  "prompt": "Remove the white sheep on the left side of the video.",
  "video": "path/to/source.mp4",
  "output": "output/edited.mp4"
}

Task types (task_type):

  • t2i: text → image
  • i2i: image editing
  • t2v: text → video
  • v2v: video editing
  • rv2v: reference image-guided video editing
  • r2v: reference image → video generation

Hands-On Commands for 6 Task Types

1. Text-to-Image (t2i) — Single GPU

python infer_single_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --case assets/testcases/t2i/t2i.json \
    --num_frames 1 \
    --guidance_mode t2v_apg

Or pass parameters directly:

python infer_single_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --prompt "A futuristic cityscape at dusk, cinematic lighting, 8K" \
    --task_type t2i \
    --num_frames 1 \
    --output output/city.png

2. Image Editing (i2i) — Single GPU

python infer_single_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --case assets/testcases/i2i/i2i.json \
    --num_frames 1 \
    --guidance_mode t2v_apg

3. Text-to-Video (t2v) — Multi-GPU

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/t2v/t2v.json \
    --guidance_mode t2v_apg

Default output: 480p / 16fps / 81 frames (approximately 5 seconds)

4. Video Editing (v2v) — Multi-GPU

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/v2v/v2v_case1.json \
    --guidance_mode v2v_apg

Example case file (weather change):

{
  "task_type": "v2v",
  "guidance_mode": "v2v_apg",
  "prompt": "Convert the video into an immersive snowy winter wonderland.",
  "video": "assets/source_videos/forest.mp4",
  "output": "output/winter.mp4"
}

5. Reference Image-Guided Editing (rv2v) — Multi-GPU

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/rv2v/rv2v_case1.json \
    --guidance_mode rv2v_apg

Best for: replacing objects, materials, weather, or styles in a video using a reference image.

6. Reference-to-Video Generation (r2v) — Up to 5 Reference Images

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/r2v/r2v_case1.json \
    --guidance_mode r2v_apg

Batch run using scripts:

# Run all tasks at once (reads BERNINI_R_CONFIG environment variable)
export BERNINI_R_CONFIG=./pretrained_models/Bernini-R-Diffusers
export NPROC_PER_NODE=8
export ULYSSES=8

bash scripts/bernini_r/run_t2i.sh
bash scripts/bernini_r/run_t2v.sh
bash scripts/bernini_r/run_v2v.sh
bash scripts/bernini_r/run_rv2v.sh

Gradio Visual Interface

# Single GPU (image tasks only)
python gradio_demo.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --port 7860

# 8-GPU parallel (video tasks)
torchrun --nproc-per-node 8 gradio_demo.py \
    --ulysses 8 \
    --config pretrained_models/Bernini-R-Diffusers \
    --port 7860 \
    --share   # generate a public URL

Enabling --use_pe enhances prompts via any OpenAI-compatible endpoint, significantly improving generation quality.

export BERNINI_PE_API_KEY=your_key
export BERNINI_PE_BASE_URL=https://api.openai.com/v1   # or Ollama/vLLM endpoint
export BERNINI_PE_MODEL=gpt-4o-mini                     # any vision model

torchrun --nproc-per-node 8 infer_multi_gpu.py \
    --config pretrained_models/Bernini-R-Diffusers \
    --ulysses 8 \
    --case assets/testcases/t2v/t2v.json \
    --use_pe

Paired with a local model (free, fully offline):

# Start Ollama
ollama serve &
ollama pull qwen2.5vl:7b

export BERNINI_PE_API_KEY=ollama
export BERNINI_PE_BASE_URL=http://localhost:11434/v1
export BERNINI_PE_MODEL=qwen2.5vl:7b

Full Pipeline (Bernini 7B+14B) Exclusive Commands

export BERNINI_CONFIG=./pretrained_models/Bernini-Diffusers
export NPROC_PER_NODE=8
export ULYSSES=8

# Text-to-video (stronger instruction following)
bash scripts/bernini/run_t2v.sh

# Complex video editing (where MLLM semantic planning has the clearest advantage)
CASE_PATH=assets/testcases/v2v/v2v_case2.json \
bash scripts/bernini/run_v2v.sh

# Gradio interface
torchrun --nproc-per-node 8 gradio_demo.py \
    --ulysses 8 \
    --config ByteDance/Bernini-Diffusers \
    --port 7860 --share

VRAM Configuration Reference

ModelGPU ConfigResolutionNotes
Bernini-R 1.3BSingle RTX 4090 (24GB)480pCommunity-verified
Bernini-R 14B8×A100 (80GB)480p/720pOfficially recommended
Bernini-R 14B4×A100 (80GB)480pReduce to --ulysses 4
Bernini-R 14B8×H100 (80GB)480p/720pOptimal, FlashAttn-3
Bernini 7B+14B8×H100 (80GB)480p/720pRecommended for full pipeline

A100/H100 compute rental available in China: AutoDL, Vast.ai, Lepton.ai (choose as needed)

Training (Fine-tune Bernini-R)

Training code fully released on 2026-07-13:

# Recommended: use uv to manage the training environment
curl -LsSf https://astral.sh/uv/install.sh | sh

uv sync
uv sync --extra all
uv pip install --no-build-isolation "flash-attn==2.8.3"

# Start training
uv run python train_bernini_r.py \
    --config configs/bernini_renderer_wan22/config.json \
    --dataset_path /path/to/your/dataset \
    --output_dir output/finetuned

Full training documentation: docs/bernini_r_train.md

Key Technical Design

Why “plan semantics” rather than “retrieve images” vector-database-style?

Traditional video editing methods feed the source video and prompt directly to a diffusion model, relying on the model to “guess” the correct editing direction. For simple style transfer this works fine, but complex instructions (“turn the right half of the video into a Russian classical court dance clay-animation stop-motion style, while keeping the left half as the original war scene”) are very hard to nail in a single step.

Bernini’s solution: first have the MLLM plan “where the target semantic embeddings should be,” then have the DiT denoise along the semantic gradient. This decomposes a hard reasoning problem into two simpler subproblems.

What problem does SA-3D RoPE solve?

Multi-source inputs (source video frames + reference image 1 + reference image 2 + target placeholder) can cause confusion in the attention mechanism. SA-3D RoPE assigns different positional encodings to tokens from different visual segments, allowing the renderer to know which tokens come from the source, which are references, and which are the target to be generated.

Comparison with Similar Open-Source Models

ModelOrganizationPlannerRendererStrongest TaskSingle-GPU Capable
BerniniByteDanceQwen2.5-VL-7BWan2.2-14BComplex video editing1.3B version only
Wan2.2AlibabaNoneMoE-14BT2V generation5B version at 720P
HunyuanVideoTencentNone13BT2V generationPartial support
CogVideoXZhipu AINone5B/13BT2V generation5B can run single-GPU

Bernini is currently the strongest open-source model for video editing (v2v) in the ecosystem, though pure T2V generation is not its focus (VBench 84.37 is slightly below Wan2.2’s top performance).

Getting Started Recommendations

  1. Start with Bernini-R 1.3B: runs on a single 24GB card — first verify that the i2i (image editing) task pipeline works end-to-end
  2. Use the Gradio interface for testing: --share generates a public URL without needing a local UI
  3. v2v is the killer use case: weather transformation, style transfer, and object removal are where Bernini shines brightest
  4. Connect a Prompt Enhancer: pair with local Ollama (Qwen2.5-VL) — the difference in prompt quality is very significant
  5. For complex instructions, use the full Bernini 7B+14B: the 1.3B model is noticeably weaker than the 14B for complex tasks such as “character motion generation”

Reference Resources

© 2026 Author: Mycelium Protocol

相关文章

Tech-Experiment

Duix-Avatar（前 HeyGem）：本地离线数字人视频生成深度拆解
Tech-Experiment

DGX Spark 上跑视频生成：FlashAttention 没用、torch.compile 没用、量化也没用
Tech-Experiment

FLUX.3 深度解析：图像+视频+音频+机器人动作，一个模型，本地部署完整指南与硬件成本评估

💬 评论与讨论

使用 GitHub 账号登录后发表评论

关于本站 · 免责声明

🍄 Mushroom Research Blog 是非营利、免费公开的个人科技观察博客与公众号 XStack18，不接受商业合作、不代表任何企业或机构立场，也不谋求商业利益。我们以个人视角客观中立地记录和分析 AI、Web3 等领域的最新模型发布与技术动态——不止转述新闻标题或二手信息，而是给出有独立思考的深入分析，希望帮更多人获得有价值的一手科技认知。

⚠️ 文中介绍的开源代码与模型，仅供学习交流与技术借鉴。它们大多仍处于早期阶段，有待进一步研究和验证，请勿直接用于工作或生产环境；如需采用，请先自行充分测试，并核实其许可证与安全性。
Open-source code and models featured here are shared for learning and reference only. Most are early-stage and still need further study and verification — please don't use them directly in your work or in production. Test them thoroughly and check their licenses and security first.

  1. 本站文章均为作者基于公开信息的个人研究与观点整理，不代表文中提及的任何公司、产品、模型的官方立场，未与其构成商业关联或合作关系。
  2. 科技行业信息更新极快，我们尽力保证内容准确、及时，但不对完整性、实时性做绝对保证，具体请以相关企业/项目官方公告为准。
  3. 文中引用的第三方商标、产品名称、图片、数据等版权归原权利人所有，我们会尽量注明来源；如你认为存在版权疑问或侵权，请通过下方邮箱联系我们，收到通知后会尽快核实处理（更正、加注来源或删除）。
  4. 文章内容仅为技术科普与个人观点，不构成投资、法律或其他专业建议，据此进行任何决策的后果需自行判断和承担。

📮 侵权 / 勘误 / 合作咨询：hello@mushroom.cv