全量训练MiniCPM-o问题

[JACKYLUO1991]

Reminder

• I have read the above rules and searched the existing issues.

System Info

训练显示：

[rank1]: File "/Data1/home/.cache/huggingface/modules/transformers_modules/MiniCPM-o-2_6/modeling_minicpmo.py", line 623, in forward
[rank1]: vllm_embedding = self.get_omni_embedding(
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/Data1/home/.cache/huggingface/modules/transformers_modules/MiniCPM-o-2_6/modeling_minicpmo.py", line 577, in get_omni_embedding
[rank1]: audio_embeddings = self.get_audio_embedding(data, chunk_length)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/Data1/home/.cache/huggingface/modules/transformers_modules/MiniCPM-o-2_6/modeling_minicpmo.py", line 493, in get_audio_embedding
[rank1]: audio_feature_lens = torch.hstack(audio_feature_lens_raw)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: TypeError: expected Tensor as element 1 in argument 0, but got list

请问如何处理呢？

Reproduction

Put your message here.

Others

No response

[BUAADreamer]

是语音微调嘛，贴一下你的yaml

[JACKYLUO1991]

@BUAADreamer
不是的，是微调视频，但是显示其中语音部分的代码有问题：

### model
model_name_or_path: /home/MiniCPM-o-2_6
trust_remote_code: true
freeze_vision_tower: true
print_param_status: true
flash_attn: fa2

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: examples/deepspeed/ds_z2_config.json

### dataset
dataset: mllm_video_demo # mllm_demo mllm_video_demo
template: minicpm_v
cutoff_len: 3072
# max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/minicpmo_2_6/full/sft
logging_steps: 1
save_steps: 100
plot_loss: true
overwrite_output_dir: true
save_total_limit: 3

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 4.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
save_only_model: true

val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 100

### eval
do_eval: false

另外，单独全量微调语音是没问题的！

[JACKYLUO1991]

   def get_audio_embedding_streaming(self, data):
        r"""
        Extract audio embeddings in a streaming manner using cached key-value pairs.
        This method processes incoming audio features incrementally and stores/updates `past_key_values`
        for faster inference on subsequent audio frames. It only supports batch_size=1 and is intended
        for streaming scenarios.
        Args:
            data (dict):
                - **"audio_features"** (`torch.FloatTensor`): Input mel-spectrograms of shape `(batch_size, 80, frames)`.
                - **"audio_feature_lens"** (List[List[int]]): Lengths of each audio segment for each item in the batch.
        Returns:
            List[List[torch.Tensor]]: audio embeddings
        """
        wavforms = data.get("audio_features", [])  # (bs, 80, frames) or [], multi audios need filled in advance
        audio_feature_lens_raw = data.get("audio_feature_lens", [])  # list, [[x1, x2], [y1], [z1]]
        
        # TODO：将列表中每个元素转换为 Tensor（如果尚未转换的话）
        audio_feature_lens_raw = [
            torch.tensor(x, device=wavforms.device) if not isinstance(x, torch.Tensor) else x
            for x in audio_feature_lens_raw
        ]

        # exist audio
        if len(wavforms) > 0:
            audio_feature_lens = torch.hstack(audio_feature_lens_raw)
            batch_size, _, max_mel_seq_len = wavforms.shape
            assert batch_size == 1
            max_seq_len = (max_mel_seq_len - 1) // 2 + 1

            if self.audio_past_key_values is not None:
                cache_length = self.audio_past_key_values[0][0].shape[2]
                apm_max_len = self.apm.embed_positions.weight.shape[0]
                if cache_length + max_seq_len >= apm_max_len:
                    logger.warning(
                        f"audio_past_key_values length {cache_length + max_seq_len} exceed {apm_max_len}, reset."
                    )
                    self.audio_past_key_values = None

            audio_outputs = self.apm(wavforms, past_key_values=self.audio_past_key_values, use_cache=True)
            audio_states = audio_outputs.last_hidden_state  # [:, :audio_feat_lengths, :]
            self.audio_past_key_values = audio_outputs.past_key_values

            audio_embeds = self.audio_projection_layer(audio_states)

            audio_embeds = audio_embeds.transpose(1, 2)
            audio_embeds = self.audio_avg_pooler(audio_embeds)
            audio_embeds = audio_embeds.transpose(1, 2)

            _, feature_lens_after_pooling = self._get_feat_extract_output_lengths(audio_feature_lens)

            num_audio_tokens = feature_lens_after_pooling

            final_audio_embeds = []
            idx = 0
            for i in range(len(audio_feature_lens_raw)):
                target_audio_embeds = []
                for _ in range(len(audio_feature_lens_raw[i])):
                    target_audio_embeds.append(audio_embeds[idx, : num_audio_tokens[idx], :])
                    idx += 1
                final_audio_embeds.append(target_audio_embeds)
            return final_audio_embeds
        else:
            return []

    def get_audio_embedding(self, data, chunk_length=-1, dummy=True):
        r"""
        Extract full audio embeddings with optional chunk-based attention.
        This method computes embeddings for all audio frames at once, either using full attention (when
        `chunk_length` is -1) or chunk-based attention (when `chunk_length` is a positive number). It does
        not use key-value caching and is suitable for non-streaming inference.
        Args:
            data (dict):
                - **"audio_features"** (`torch.FloatTensor`): Input mel-spectrograms of shape `(batch_size, 80, frames)`.
                - **"audio_feature_lens"** (List[List[int]]): Lengths of each audio segment for each item in the batch.
            chunk_length (int, optional): Determines whether to use full attention (-1) or chunk-based
                attention (>0) during embedding computation.
        Returns:
            List[List[torch.Tensor]]: audio embeddings
        """
        
        wavforms = data.get("audio_features", [])  # (bs, 80, frames) or [], multi audios need filled in advance
        audio_feature_lens_raw = data.get("audio_feature_lens", [])  # list, [[x1, x2], [y1], [z1]]

        # exist audio
        if len(wavforms) > 0:
            # TODO：添加额外判断
            audio_feature_lens_raw = [
                torch.tensor(x, device=wavforms.device) if not isinstance(x, torch.Tensor) else x
                for x in audio_feature_lens_raw
            ]
            audio_feature_lens = torch.hstack(audio_feature_lens_raw)
            batch_size, _, max_mel_seq_len = wavforms.shape
            max_seq_len = (max_mel_seq_len - 1) // 2 + 1

            # Create a sequence tensor of shape (batch_size, max_seq_len)
            seq_range = (
                torch.arange(0, max_seq_len, dtype=audio_feature_lens.dtype, device=audio_feature_lens.device)
                .unsqueeze(0)
                .expand(batch_size, max_seq_len)
            )
            lengths_expand = audio_feature_lens.unsqueeze(1).expand(batch_size, max_seq_len)
            # Create mask
            padding_mask = seq_range >= lengths_expand  # 1 for padded values

            audio_attention_mask_ = padding_mask.view(batch_size, 1, 1, max_seq_len).expand(
                batch_size, 1, max_seq_len, max_seq_len
            )
            audio_attention_mask = audio_attention_mask_.to(
                dtype=self.apm.conv1.weight.dtype, device=self.apm.conv1.weight.device
            )

            if chunk_length > 0:
                chunk_num_frame = int(chunk_length * 50)
                chunk_mask = self.subsequent_chunk_mask(
                    size=max_seq_len,
                    chunk_size=chunk_num_frame,
                    num_left_chunks=-1,
                    device=audio_attention_mask_.device,
                )
                audio_attention_mask_ = torch.logical_or(audio_attention_mask_, torch.logical_not(chunk_mask))

            audio_attention_mask[audio_attention_mask_] = float("-inf")
            audio_states = self.apm(
                wavforms, output_hidden_states=True, attention_mask=audio_attention_mask
            ).hidden_states[self.audio_encoder_layer]
            audio_embeds = self.audio_projection_layer(audio_states)

            audio_embeds = audio_embeds.transpose(1, 2)
            audio_embeds = self.audio_avg_pooler(audio_embeds)
            audio_embeds = audio_embeds.transpose(1, 2)

            _, feature_lens_after_pooling = self._get_feat_extract_output_lengths(audio_feature_lens)

            num_audio_tokens = feature_lens_after_pooling

            final_audio_embeds = []
            idx = 0
            for i in range(len(audio_feature_lens_raw)):
                target_audio_embeds = []
                for _ in range(len(audio_feature_lens_raw[i])):
                    target_audio_embeds.append(audio_embeds[idx, : num_audio_tokens[idx], :])
                    idx += 1
                final_audio_embeds.append(target_audio_embeds)
            return final_audio_embeds
        elif self.training and dummy:
            dtype = self.apm.embed_positions.weight.dtype
            device = self.apm.embed_positions.weight.device

            dummy_wavs = torch.zeros((1, 80, 100), device=device, dtype=dtype)
            audio_states = self.apm(dummy_wavs, output_hidden_states=True).hidden_states[self.audio_encoder_layer]

            audio_embeds = self.audio_projection_layer(audio_states)

            audio_embeds = audio_embeds.transpose(1, 2)
            audio_embeds = self.audio_avg_pooler(audio_embeds)
            audio_embeds = audio_embeds.transpose(1, 2)
            return [audio_embeds]

        else:
            return []
    def get_omni_embedding(self, data, input_embeddings, chunk_length=-1, stream_input=False):
        """
        Args:
            data:
            input_embeddings:
            chunk_length: whisper use full attention or chunk attention
            stream_input: use streaming audio embedding
        Returns:
            final embeddings with audio feature
        """
        if stream_input:
            audio_embeddings = self.get_audio_embedding_streaming(data)
        else:
            audio_embeddings = self.get_audio_embedding(data, chunk_length)
    
        bs = len(input_embeddings)
        if len(data.get("audio_features", [])) > 0:
            assert len(audio_embeddings) == len(input_embeddings)
            if len(audio_embeddings) > 0:
                audio_bounds = data["audio_bounds"]
    
                if self.config.chunk_input:
                    for i in range(bs):
                        # 如果当前样本的音频嵌入列表为空，则跳过或进行其他处理
                        if not audio_embeddings[i]:
                            # logger.warning(f"第 {i} 个样本的音频嵌入为空，跳过音频融合。")
                            continue
    
                        audio_embs = torch.cat(audio_embeddings[i], dim=0).to(
                            device=input_embeddings.device, dtype=input_embeddings.dtype
                        )
                        audio_start_pos = 0
                        for bound in audio_bounds[i]:
                            audio_len = bound[1] - bound[0]
                            input_embeddings[i, bound[0]:bound[1]] = audio_embs[
                                audio_start_pos: audio_start_pos + audio_len, :
                            ]
                            audio_start_pos += audio_len
                else:
                    for i in range(bs):
                        audio_embs = audio_embeddings[i]
                        bounds = audio_bounds[i]
                        for embs, bound in zip(audio_embs, bounds):
                            audio_indices = torch.arange(bound[0], bound[1], dtype=torch.long).to(
                                input_embeddings.device
                            )
                            if embs.shape[0] != len(audio_indices):
                                raise ValueError(
                                    f"Shape mismatch: Trying to assign embeddings of shape {embs.shape} "
                                    f"to input indices of length {len(audio_indices)}"
                                )
                            input_embeddings[i, audio_indices] = embs.to(input_embeddings.dtype)
        elif self.training:
            for i in range(bs):
                # dummy audio_embeddings
                input_embeddings[i] = input_embeddings[i] + 0
        return input_embeddings

我修改了上述函数部分代码，现在可以正常运行了。

[BUAADreamer]

就是修改了这一行是吧

audio_feature_lens_raw = [
                torch.tensor(x, device=wavforms.device) if not isinstance(x, torch.Tensor) else x
                for x in audio_feature_lens_raw
            ]

[JACKYLUO1991]

还有

  if not audio_embeddings[i]:
      continue

[MengHao666]

用MiniCPM-o和MiniCPM-V模型，全量微调视频，出现同样的问题，不知道你有没有解决

[MengHao666]

get_audio_embedding_streaming

请问是transformers库的问题吗

[BUAADreamer]

@MengHao666 @JACKYLUO1991 这个PR已解决minicpm-o相关问题：#6890

这个修改参考了部分 @JACKYLUO1991 的解决方案，感谢！！

需要从huggingface更新一下 modeling_minicpmo文件（在MiniCPM-o-2.6模型目录中）

以及，现在需要分别使用minicpm_o和minicpm_v模板，对于这两个模型

[MengHao666]

@MengHao666 @JACKYLUO1991 这个PR已解决minicpm-o相关问题：#6890

这个修改参考了部分 @JACKYLUO1991 的解决方案，感谢！！

需要更新一下 modeling_minicpmo文件中（在MiniCPM-o-2.6模型目录中）

thanks， hope this feature could be merged into main branch soon.