fix id change bug

Author: TianQi-777

README.md

@@ -42,6 +42,7 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
 
 
 ## 🔥🔥🔥 News!!
+* Mar 07, 2025: 🔥 We have fixed the bug in our open-source version that caused ID changes. Please try the new model weights of [HunyuanVideo-I2V](https://huggingface.co/tencent/HunyuanVideo-I2V) to ensure full visual consistency in the first frame and produce higher quality videos. 
 * Mar 06, 2025: 👋 We release the inference code and model weights of HunyuanVideo-I2V. [Download](https://github.com/Tencent/HunyuanVideo-I2V/blob/main/ckpts/README.md).
 
 
@@ -53,7 +54,12 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
   <p>Co-creator @D-aiY Director Ding Yi</p>
 </div>
 
-### Customizable I2V LoRA Demo
+### Frist Frame Consistency Demo
+|  Reference Image | Generated Video  |Reference Image | Generated Video  |Reference Image | Generated Video  |
+|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|
+|        <img src="https://github.com/user-attachments/assets/83e7a097-ffca-40db-9c72-be01d866aa7d" width="80%">         |       <video src="https://github.com/user-attachments/assets/f81d2c88-bb1a-43f8-b40f-1ccc20774563" width="100%"> </video>        |       <img src="https://github.com/user-attachments/assets/c385a11f-60c7-4919-b0f1-bc5e715f673c" width="50%">         |       <video src="https://github.com/user-attachments/assets/0c29ede9-0481-4d40-9c67-a4b6267fdc2d" width="100%"> </video>        | <img src="https://github.com/user-attachments/assets/5763f5eb-0be5-4b36-866a-5199e31c5802" width="95%">         |       <video src="https://github.com/user-attachments/assets/a8da0a1b-ba7d-45a4-a901-5d213ceaf50e" width="100%"> </video>        |
+
+<!-- ### Customizable I2V LoRA Demo
 
 | I2V Lora Effect |  Reference Image | Generated Video  |
 |:---------------:|:--------------------------------:|:----------------:|
@@ -74,16 +80,16 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
 - Enhance-A-Video (Better Generated Video for Free): [Enhance-A-Video](https://github.com/NUS-HPC-AI-Lab/Enhance-A-Video) by [NUS-HPC-AI-Lab](https://ai.comp.nus.edu.sg/)
 - TeaCache (Cache-based Accelerate): [TeaCache](https://github.com/LiewFeng/TeaCache) by [Feng Liu](https://github.com/LiewFeng)
 - HunyuanVideoGP (GPU Poor version): [HunyuanVideoGP](https://github.com/deepbeepmeep/HunyuanVideoGP) by [DeepBeepMeep](https://github.com/deepbeepmeep)
- -->
+ --> 
 
 
 
 ## 📑 Open-source Plan
 - HunyuanVideo-I2V (Image-to-Video Model)
-  - [x] Lora training scripts
   - [x] Inference 
   - [x] Checkpoints
   - [x] ComfyUI
+  - [ ] Lora training scripts
   - [ ] Multi-gpus Sequence Parallel inference (Faster inference speed on more gpus)
   - [ ] Diffusers 
   - [ ] FP8 Quantified weight
@@ -93,7 +99,7 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
   - [🔥🔥🔥 News!!](#-news)
   - [🎥 Demo](#-demo)
     - [I2V Demo](#i2v-demo)
-    - [Customizable I2V LoRA Demo](#customizable-i2v-lora-demo)
+    - [Frist Frame Consistency Demo](#frist-frame-consistency-demo)
   - [📑 Open-source Plan](#-open-source-plan)
   - [Contents](#contents)
   - [**HunyuanVideo-I2V Overall Architecture**](#hunyuanvideo-i2v-overall-architecture)
@@ -105,18 +111,12 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
     - [Tips for Using Image-to-Video Models](#tips-for-using-image-to-video-models)
     - [Using Command Line](#using-command-line)
     - [More Configurations](#more-configurations)
-  - [🎉 Customizable I2V LoRA effects training](#-customizable-i2v-lora-effects-training)
-    - [Requirements](#requirements)
-    - [Environment](#environment)
-    - [Training data construction](#training-data-construction)
-    - [Training](#training)
-    - [Inference](#inference)
   - [🔗 BibTeX](#-bibtex)
   - [Acknowledgements](#acknowledgements)
 ---
 
 ## **HunyuanVideo-I2V Overall Architecture**
-Leveraging the advanced video generation capabilities of [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), we have extended its application to image-to-video generation tasks. To achieve this, we employ an image latent concatenation technique to effectively reconstruct and incorporate reference image information into the video generation process.
+Leveraging the advanced video generation capabilities of [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), we have extended its application to image-to-video generation tasks. To achieve this, we employ a token replace technique to effectively reconstruct and incorporate reference image information into the video generation process.
 
 Since we utilizes a pre-trained Multimodal Large Language Model (MLLM) with a Decoder-Only architecture as the text encoder, we can significantly enhance the model's ability to comprehend the semantic content of the input image and to seamlessly integrate information from both the image and its associated caption. Specifically, the input image is processed by the MLLM to generate semantic image tokens. These tokens are then concatenated with the video latent tokens, enabling comprehensive full-attention computation across the combined data.
 
@@ -212,12 +212,6 @@ Similar to [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), HunyuanVideo
   - **Camera Angle (Optional)**: Indicate the perspective or viewpoint.
 - **Avoid Overly Detailed Prompts**: Lengthy or highly detailed prompts can lead to unnecessary transitions in the video output.
 
-For example:
-1. A man with short gray hair plays a red electric guitar.
-2. A woman sits on a wooden floor, holding a colorful bag.
-3. A bee flaps its wings. The camera movement is Zoom Out/Zoom In/Pan Right.
-4. A little boy closes his mouth, stands up, and lifts his left hand. The background is blurred.
-
 <!-- **For image-to-video models, we recommend using concise prompts to guide the model's generation process. A good prompt should include elements such as background, main subject, action, and camera angle. Overly long or excessively detailed prompts may introduce unnecessary transitions.** -->
 
 ### Using Command Line
@@ -266,7 +260,7 @@ We list some more useful configurations for easy usage:
 |     `--save-path`      |         ./results             |         Path to save the generated video.                     |
 
 
-## 🎉 Customizable I2V LoRA effects training
+<!-- ## 🎉 Customizable I2V LoRA effects training
 
 ### Requirements
 
@@ -336,7 +330,7 @@ We list some lora specific configurations for easy usage:
 |:-------------------:|:-------:|:----------------------------:|
 |    `--use-lora`     |  False  |  Whether to open lora mode.  |
 |   `--lora-scale`    |   1.0   | Fusion scale for lora model. |
-|   `--lora-path`     |   ""    |  Weight path for lora model. |
+|   `--lora-path`     |   ""    |  Weight path for lora model. | -->
 
 
 ## 🔗 BibTeX

README_zh.md

@@ -34,6 +34,7 @@
 > [**HunyuanVideo: A Systematic Framework For Large Video Generation Model**](https://arxiv.org/abs/2412.03603)
 
 ## 🔥🔥🔥 最新动态
+* 2025年3月7日: 🔥 我们已经修复了开源版本中导致ID变化的bug，请尝试[HunyuanVideo-I2V](https://huggingface.co/tencent/HunyuanVideo-I2V)新的模型权重，以确保首帧完全视觉一致性，并制作更高质量的视频。
 * 2025年3月6日: 👋 发布HunyuanVideo-I2V的推理代码和模型权重。[下载地址](https://github.com/Tencent/HunyuanVideo-I2V/blob/main/ckpts/README.md)
 
 ## 🎥 演示
@@ -43,19 +44,24 @@
   <p>联合创作 @D-aiY 导演 丁一</p>
 </div>
 
-### 定制化I2V LoRA效果演示
+### 首帧一致性示例
+|  参考图 | 生成视频  |参考图 | 生成视频   |参考图 | 生成视频  |
+|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|
+|        <img src="https://github.com/user-attachments/assets/83e7a097-ffca-40db-9c72-be01d866aa7d" width="80%">         |       <video src="https://github.com/user-attachments/assets/f81d2c88-bb1a-43f8-b40f-1ccc20774563" width="100%"> </video>        |       <img src="https://github.com/user-attachments/assets/c385a11f-60c7-4919-b0f1-bc5e715f673c" width="50%">         |       <video src="https://github.com/user-attachments/assets/0c29ede9-0481-4d40-9c67-a4b6267fdc2d" width="100%"> </video>        | <img src="https://github.com/user-attachments/assets/5763f5eb-0be5-4b36-866a-5199e31c5802" width="95%">         |       <video src="https://github.com/user-attachments/assets/a8da0a1b-ba7d-45a4-a901-5d213ceaf50e" width="100%"> </video>        |
+
+<!-- ### 定制化I2V LoRA效果演示
 
 | 特效类型       |  参考图像  | 生成视频  |
 |:---------------:|:--------------------------------:|:----------------:|
 |   头发生长   |        <img src="./assets/demo/i2v_lora/imgs/hair_growth.png" width="40%">         |       <video src="https://github.com/user-attachments/assets/06b998ae-bbde-4c1f-96cb-a25a9197d5cb" width="100%"> </video>        |
-|     拥抱     |      <img src="./assets/demo/i2v_lora/imgs/embrace.png" width="40%">          |       <video src="https://github.com/user-attachments/assets/f8c99eb1-2a43-489a-ba02-6bd50a6dd260" width="100%" > </video>        |
+|     拥抱     |      <img src="./assets/demo/i2v_lora/imgs/embrace.png" width="40%">          |       <video src="https://github.com/user-attachments/assets/f8c99eb1-2a43-489a-ba02-6bd50a6dd260" width="100%" > </video>        | -->
 
 ## 📑 开源计划
 - HunyuanVideo-I2V（图像到视频模型）
-  - [x] LoRA训练脚本
   - [x] 推理代码
   - [x] 模型权重
   - [x] ComfyUI支持
+  - [ ] LoRA训练脚本
   - [ ] 多GPU序列并行推理（提升多卡推理速度）
   - [ ] Diffusers集成 
   - [ ] FP8量化权重
@@ -65,7 +71,7 @@
   - [🔥🔥🔥 最新动态](#-最新动态)
   - [🎥 演示](#-演示)
     - [I2V 示例](#i2v-示例)
-    - [定制化I2V LoRA效果演示](#定制化i2v-lora效果演示)
+    - [首帧一致性示例](#首帧一致性示例)
   - [📑 开源计划](#-开源计划)
   - [目录](#目录)
   - [**HunyuanVideo-I2V 整体架构**](#hunyuanvideo-i2v-整体架构)
@@ -77,19 +83,13 @@
     - [使用图生视频模型的建议](#使用图生视频模型的建议)
     - [使用命令行](#使用命令行)
     - [更多配置](#更多配置)
-  - [🎉自定义 I2V LoRA 效果训练](#自定义-i2v-lora-效果训练)
-    - [要求](#要求)
-    - [训练环境](#训练环境)
-    - [训练数据构建](#训练数据构建)
-    - [开始训练](#开始训练)
-    - [推理](#推理)
   - [🔗 BibTeX](#-bibtex)
   - [致谢](#致谢)
 
 ---
 
 ## **HunyuanVideo-I2V 整体架构**
-基于[HunyuanVideo](https://github.com/Tencent/HunyuanVideo)强大的视频生成能力，我们将其扩展至图像到视频生成任务。为此，我们采用图像隐空间拼接技术，有效重构并融合参考图像信息至视频生成流程中。
+基于[HunyuanVideo](https://github.com/Tencent/HunyuanVideo)强大的视频生成能力，我们将其扩展至图像到视频生成任务。为此，我们采用首帧Token替换方案，有效重构并融合参考图像信息至视频生成流程中。
 
 由于我们使用预训练的Decoder-Only架构多模态大语言模型（MLLM）作为文本编码器，可用于显著增强模型对输入图像语义内容的理解能力，并实现图像与文本描述信息的深度融合。具体而言，输入图像经MLLM处理后生成语义图像tokens，这些tokens与视频隐空间tokens拼接，实现跨模态的全注意力计算。
 
@@ -224,7 +224,7 @@ python3 sample_image2video.py \
 |     `--save-path`      |         ./results             |         保存生成视频的路径。                     |
 
 
-## 🎉自定义 I2V LoRA 效果训练
+<!-- ## 🎉自定义 I2V LoRA 效果训练
 
 ###  要求
 
@@ -292,7 +292,7 @@ python3 sample_image2video.py \
 |:-------------------:|:-------:|:----------------------------:|
 |    `--use-lora`     |  None   |  是否开启 LoRA 模式。  |
 |   `--lora-scale`    |   1.0   | LoRA 模型的融合比例。 |
-|   `--lora-path`     |   ""    |  LoRA 模型的权重路径。 |
+|   `--lora-path`     |   ""    |  LoRA 模型的权重路径。 | -->
 
 ## 🔗 BibTeX
 

assets/backbone.png

@@ -504,15 +504,36 @@ def add_i2v_args(parser: argparse.ArgumentParser):
     group = parser.add_argument_group(title="I2V args")
 
     group.add_argument(
-        "--i2v-mode", action="store_true", help="Whether to open i2v mode."
+        "--i2v-mode",
+        action="store_true",
+        help="Whether to open i2v mode."
+    )
+
+    group.add_argument(
+        "--i2v-resolution",
+        type=str,
+        default="720p",
+        choices=["720p", "540p", "360p"],
+        help="Resolution for i2v inference."
     )
 
     group.add_argument(
-        "--i2v-resolution", type=str, default="720p", choices=["720p", "540p", "360p"], help="Resolution for i2v inference."
+        "--i2v-image-path",
+        type=str,
+        default="./assets/demo/i2v/imgs/0.png",
+        help="Image path for i2v inference."
+    )
+
+    group.add_argument(
+        "--i2v-condition-type",
+        type=str,
+        default="token_replace",
+        choices=["token_replace", "latent_concat"],
+        help="Condition type for i2v model."
     )
 
     group.add_argument(
-        "--i2v-image-path", type=str, default="./assets/demo/i2v/imgs/0.png", help="Image path for i2v inference."
+        "--i2v-stability", action="store_true", help="Whether to use i2v stability mode."
     )
 
     return parser

assets/demo/i2v/imgs/0.jpg

@@ -576,8 +576,10 @@ def prepare_latents(
         latents=None,
         img_latents=None,
         i2v_mode=False,
+        i2v_condition_type=None,
+        i2v_stability=True,
     ):
-        if i2v_mode:
+        if i2v_mode and i2v_condition_type == "latent_concat":
             num_channels_latents = (num_channels_latents - 1) // 2
         shape = (
             batch_size,
@@ -592,7 +594,7 @@ def prepare_latents(
                 f" size of {batch_size}. Make sure the batch size matches the length of the generators."
             )
 
-        if i2v_mode:
+        if i2v_mode and i2v_stability:
             if img_latents.shape[2] == 1:
                 img_latents = img_latents.repeat(1, 1, video_length, 1, 1)
             x0 = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
@@ -722,6 +724,8 @@ def __call__(
         n_tokens: Optional[int] = None,
         embedded_guidance_scale: Optional[float] = None,
         i2v_mode: bool = False,
+        i2v_condition_type: str = None,
+        i2v_stability: bool = True,
         img_latents: Optional[torch.Tensor] = None,
         semantic_images=None,
         **kwargs,
@@ -963,9 +967,11 @@ def __call__(
             latents,
             img_latents=img_latents,
             i2v_mode=i2v_mode,
+            i2v_condition_type=i2v_condition_type,
+            i2v_stability=i2v_stability
         )
 
-        if i2v_mode:
+        if i2v_mode and i2v_condition_type == "latent_concat":
             if img_latents.shape[2] == 1:
                 img_latents_concat = img_latents.repeat(1, 1, video_length, 1, 1)
             else:
@@ -1004,8 +1010,11 @@ def __call__(
                 if self.interrupt:
                     continue
 
+                if i2v_mode and i2v_condition_type == "token_replace":
+                    latents = torch.concat([img_latents, latents[:, :, 1:, :, :]], dim=2)
+
                 # expand the latents if we are doing classifier free guidance
-                if i2v_mode:
+                if i2v_mode and i2v_condition_type == "latent_concat":
                     latent_model_input = torch.concat([latents, img_latents_concat, mask_concat], dim=1)
                 else:
                     latent_model_input = latents
@@ -1066,9 +1075,17 @@ def __call__(
                     )
 
                 # compute the previous noisy sample x_t -> x_t-1
-                latents = self.scheduler.step(
-                    noise_pred, t, latents, **extra_step_kwargs, return_dict=False
-                )[0]
+                if i2v_mode and i2v_condition_type == "token_replace":
+                    latents = self.scheduler.step(
+                        noise_pred[:, :, 1:, :, :], t, latents[:, :, 1:, :, :], **extra_step_kwargs, return_dict=False
+                    )[0]
+                    latents = torch.concat(
+                        [img_latents, latents], dim=2
+                    )
+                else:
+                    latents = self.scheduler.step(
+                        noise_pred, t, latents, **extra_step_kwargs, return_dict=False
+                    )[0]
 
                 if callback_on_step_end is not None:
                     callback_kwargs = {}
@@ -1139,7 +1156,7 @@ def __call__(
         # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
         image = image.cpu().float()
 
-        if i2v_mode:
+        if i2v_mode and i2v_condition_type == "latent_concat":
             image = image[:, :, 4:, :, :]
 
         # Offload all models