Repository navigation
Best Practices for Inference and Fine-Tuning with MiniCPM-V 2.6 #1613
Description
模型：https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6
通常，多模态大模型微调会使用自定义数据集进行微调。在这里，我们将展示可直接运行的demo。
在开始微调之前，请确保您的环境已准备妥当。
git clone https://github.com/modelscope/swift.git
cd swift
pip install -e .[llm]模型推理
CUDA_VISIBLE_DEVICES=0 swift infer \
--model_type minicpm-v-v2_6-chat \
--model_id_or_path OpenBMB/MiniCPM-V-2_6<<< 你好
你好！今天我能为您提供什么帮助？
--------------------------------------------------
<<< clear
<<< <image>描述这张图片
Input an image path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png
这张图片展示了一只小猫的特写，它有着引人注目的外貌。小猫有着大大的、圆圆的、蓝色的眼睛，看起来充满了好奇和天真。它的毛色主要是白色，带有灰色和黑色的条纹，特别是在脸部和耳朵周围，这些地方的条纹更加明显。小猫的耳朵竖立着，尖尖的，内侧是粉红色的。它的胡须又长又白，从脸颊上伸出来。小猫的鼻子是粉红色的，嘴巴微微张开，露出一点粉红色的舌头。背景模糊，将焦点集中在小猫身上，暗示着一个室内环境，柔和的光线照亮了小猫的毛发。
--------------------------------------------------
<<< clear
<<< <video>描述这段视频
Input a video path or URL <<< https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4
这段视频展示了一个年幼的孩子坐在床上，专心阅读一本书。孩子戴着深色眼镜，穿着浅蓝色无袖上衣和粉色裤子。床上铺着白色床单，孩子旁边放着一件白色衣物。背景中有一个木制婴儿床，暗示着一个家庭环境。房间光线柔和，氛围平静。视频中没有明显的动作或活动，孩子似乎完全沉浸在阅读中。
图片微调
我们使用 coco-en-mini 数据集进行微调，该数据集的任务是对图片内容进行描述。您可以在 modelscope 上找到该数据集：https://modelscope.cn/datasets/modelscope/coco_2014_caption/summary
# 默认会将lora_target_modules设置为llm和resampler所有的linear
CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 swift sft \
--model_type minicpm-v-v2_6-chat \
--model_id_or_path OpenBMB/MiniCPM-V-2_6 \
--sft_type lora \
--dataset coco-en-mini#20000 \
--deepspeed default-zero2如果要使用自定义数据集，只需按以下方式进行指定：
--dataset train.jsonl \
--val_dataset val.jsonl \自定义数据集支持json和jsonl样式，以下是自定义数据集的样例：
{"query": "<image>55555", "response": "66666", "images": ["image_path"]}
{"query": "eeeee<image>eeeee<image>eeeee", "response": "fffff", "history": [], "images": ["image_path1", "image_path2"]}
{"query": "EEEEE", "response": "FFFFF", "history": [["query1", "response2"], ["query2", "response2"]], "images": []}
显存占用：
微调后推理脚本如下：
# 如果要全量测试请设置: `--show_dataset_sample -1`
CUDA_VISIBLE_DEVICES=0 swift infer \
--ckpt_dir output/minicpm-v-v2_6-chat/vx-xxx/checkpoint-xxx \
--load_dataset_config true --merge_lora true微调后模型对验证集进行推理的示例（时间原因，只训练了300个step）：
视频微调
我们使用 video-chatgpt 数据集进行微调，该数据集的任务是对视频内容进行描述。您可以在 modelscope 上找到该数据集：https://modelscope.cn/datasets/swift/VideoChatGPT
CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 swift sft \
--model_type minicpm-v-v2_6-chat \
--model_id_or_path OpenBMB/MiniCPM-V-2_6 \
--sft_type lora \
--dataset video-chatgpt \
--deepspeed default-zero2自定义数据集支持json和jsonl样式，以下是自定义数据集的样例：
{"query": "<video>55555", "response": "66666", "videos": ["video_path"]}
{"query": "eeeee<video>eeeee<video>eeeee", "response": "fffff", "history": [], "videos": ["video_path1", "video_path2"]}
{"query": "EEEEE", "response": "FFFFF", "history": [["query1", "response2"], ["query2", "response2"]], "videos": []}显存占用：
微调后推理脚本如下：
CUDA_VISIBLE_DEVICES=0 swift infer \
--ckpt_dir output/minicpm-v-v2_6-chat/vx-xxx/checkpoint-xxx \
--load_dataset_config true --merge_lora true微调后模型对验证集进行推理的示例（时间原因，只训练了50个step）：
Activity
[-]MiniCPM-V 2.6的图片和视频微调[/-][+]MiniCPM-V 2.6的图片和视频微调 最佳实践[/+][-]MiniCPM-V 2.6的图片和视频微调 最佳实践[/-][+]MiniCPM-V 2.6的图片和视频微调的最佳实践[/+][-]MiniCPM-V 2.6的图片和视频微调的最佳实践[/-][+]MiniCPM-V 2.6的推理和微调最佳实践[/+]demoninpiano commentedon Aug 7, 2024
官方文档的多图理解和in-context有在swift api里支持吗？
Jintao-Huang commentedon Aug 7, 2024
支持多图和多轮的
多图需要使用多个![]()
标签即可. 可以查看上面的自定义数据集的格式
guihonghao commentedon Aug 8, 2024
需要升级swift到什么版本啊？
Jintao-Huang commentedon Aug 8, 2024
还在main分支
compleXuan commentedon Aug 11, 2024
单样本视频推理的代码可以提供吗
Jintao-Huang commentedon Aug 12, 2024
okideal commentedon Aug 12, 2024
请问官方的few-shot推理方式 swift有支持么?
yingdachen commentedon Aug 12, 2024
is this included in documentation somewhere...
Jintao-Huang commentedon Aug 12, 2024
Thank you for the excellent suggestions. We will update the document within this week.
77 remaining items
NirHeaven commentedon Oct 22, 2024
A800上使用
CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 swift sft
--model_type minicpm-v-v2_6-chat
--model_id_or_path $MODEL
--sft_type lora
--dataset coco-en-mini#20000
--deepspeed default-zero2
2013358072 commentedon Oct 28, 2024
the same question
chen10089 commentedon Nov 11, 2024
如何使用sft后的模型进行推理？
CUDA_VISIBLE_DEVICES=1 swift infer
--model_type minicpm-v-v2_6-chat
--model_id_or_path OpenBMB/MiniCPM-V-2_6
--ckpt_dir output/minicpm-v-v2_6-chat/v1-20241111-150624/checkpoint-10
是这样设置吗
chen10089 commentedon Nov 11, 2024
参考了https://swift.readthedocs.io/en/latest/Multi-Modal/minicpm-v-best-practice.html，原来是这样设置：
CUDA_VISIBLE_DEVICES=1 swift infer
--ckpt_dir output/minicpm-v-v2_6-chat/v1-20241111-150624/checkpoint-10-merged
zhaoyangwei123 commentedon Nov 13, 2024
@Jintao-Huang
利用8卡4090微调minicpm-v-v2_6-chat-int4模型报CUDA out of memory，int4模型要比正常模型要小很多呀，但是lora的时候显存持续上涨一直到超过24G，这是为什么呢？
命令行：
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NPROC_PER_NODE=8 swift sft --model_type minicpm-v-v2_6-chat-int4 --model_id_or_path OpenBMB/MiniCPM-V-2_6-int4 --sft_type lora --dataset /home/wzy/disk1/MiniCPM-V/finetune/relation_datasets/standard_train_no_history.jsonl --deepspeed zero2-offload --eval_steps 1000 --lora_dtype fp16
{'loss': 1.6643486, 'acc': 0.54588017, 'grad_norm': 4.12383413, 'learning_rate': 9.978e-05, 'memory(GiB)': 18.58, 'train_speed(iter/s)': 0.49618, 'epoch': 0.08, 'global_step/max_steps': '865/11043', 'percentage': '7.83%', 'elapsed_time': '28m 56s', 'remaining_time': '5h 40m 31s'}
{'loss': 1.60014324, 'acc': 0.59105425, 'grad_norm': 4.04182339, 'learning_rate': 9.977e-05, 'memory(GiB)': 18.58, 'train_speed(iter/s)': 0.49637, 'epoch': 0.08, 'global_step/max_steps': '870/11043', 'percentage': '7.88%', 'elapsed_time': '29m 5s', 'remaining_time': '5h 40m 14s'}
Train: 8%|████████▌ | 874/11043 [29:14<6:30:05, 2.30s/it]Traceback (most recent call last):
File "/mnt/sdb/MiniCPM-V/ms-swift/swift/cli/sft.py", line 5, in
sft_main()
File "/mnt/sdb/MiniCPM-V/ms-swift/swift/utils/run_utils.py", line 32, in x_main
result = llm_x(args, **kwargs)
File "/mnt/sdb/MiniCPM-V/ms-swift/swift/llm/sft.py", line 546, in llm_sft
return trainer_train(args, model, template, train_dataset, val_dataset, callbacks=callbacks, msg=msg)
File "/mnt/sdb/MiniCPM-V/ms-swift/swift/llm/sft.py", line 496, in trainer_train
trainer.train(training_args.resume_from_checkpoint)
File "/mnt/sdb/MiniCPM-V/ms-swift/swift/trainers/mixin.py", line 493, in train
res = super().train(resume_from_checkpoint, *args, **kwargs)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/transformers/trainer.py", line 1859, in train
return inner_training_loop(
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/transformers/trainer.py", line 2203, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/transformers/trainer.py", line 3147, in training_step
self.accelerator.backward(loss)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/accelerate/accelerator.py", line 2117, in backward
self.deepspeed_engine_wrapped.backward(loss, **kwargs)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 166, in backward
self.engine.backward(loss, **kwargs)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1976, in backward
self.optimizer.backward(loss, retain_graph=retain_graph)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 2051, in backward
self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
scaled_loss.backward(retain_graph=retain_graph)
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/home/wzy/anaconda3/envs/MiniCPM-V/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.96 GiB. GPU 0 has a total capacty of 23.65 GiB of which 7.77 GiB is free. Including non-PyTorch memory, this process has 15.87 GiB memory in use. Of the allocated memory 13.98 GiB is allocated by PyTorch, and 1.29 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
compleXuan commentedon Dec 31, 2024
您好，我目前需要进行多图推理，是否方便提供多图的单样本推理的代码。同时我需要进行大量的调用，每次调用最多20张图片，请问这会产生什么问题吗？感谢您的回答
primarybaby commentedon Jan 2, 2025
(MiniCPM-V) aipc@bogon swift % CUDA_VISIBLE_DEVICES=0 swift infer \
Jintao-Huang commentedon Feb 5, 2025
swift3, please refer to here: https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal
zhuqh19 commentedon Apr 11, 2025
请问这个问题解决了吗
dpramanik2289 commentedon Jul 3, 2025
@Jintao-Huang I need this model for video captioning. I am trying to train minicpm-v-v2_6 model but getting error -
CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 swift sft
--model_type minicpm-v-v2_6-chat
--model_id_or_path OpenBMB/MiniCPM-V-2_6
--sft_type lora
--dataset /home/pramand/Download/train2.json
--deepspeed default-zero2
why I am not able to train the model raise ValueError(f"model_type: '{model_type}' not in {list(MODEL_MAPPING.keys())}")
ValueError: model_type: 'minicpm-v-v2_6-chat' not in
why this - --model_type minicpm-v-v2_6-chat not available ? and --model_id_or_path OpenBMB/MiniCPM-V-2_6 not available its saying? any changes happened ??
dpramanik2289 commentedon Jul 3, 2025
@Jintao-Huang @yingdachen @qxzheng
please help me why getting error -
I need this model for video captioning. I am trying to train minicpm-v-v2_6 model but getting error -
CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 swift sft
--model_type minicpm-v-v2_6-chat
--model_id_or_path OpenBMB/MiniCPM-V-2_6
--sft_type lora
--dataset /home/pramand/Download/train2.json
--deepspeed default-zero2
why I am not able to train the model raise ValueError(f"model_type: '{model_type}' not in {list(MODEL_MAPPING.keys())}")
ValueError: model_type: 'minicpm-v-v2_6-chat' not in
why this - --model_type minicpm-v-v2_6-chat not available ? and --model_id_or_path OpenBMB/MiniCPM-V-2_6 not available its saying? any changes happened ??
Jintao-Huang commentedon Jul 3, 2025
长文本 主要的显存占用 不在模型。建议降低上下文长度，即降低图片分辨率
Jintao-Huang commentedon Jul 3, 2025
Here we are introducing the shell commands for swift2. For swift3, please refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal.
You only need to modify the --model parameter.
For more information, please refer to the documentation.
dpramanik2289 commentedon Jul 11, 2025
@Jintao-Huang @yingdachen @qxzheng
I have 8 GPU machine trying to do full fine tune of MiniCPM-V-2_6 But getting lots of error how to fix those error any help ??
run_command -
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
torchrun --nproc_per_node=8 swift/cli/pt.py
--model_type minicpmv2_6
--model OpenBMB/MiniCPM-V-2_6
--dataset /Download/train2.json
--streaming true
--train_type full
--output_dir output
--max_steps 5
--num_train_epochs 1
--eval_steps 2
--save_steps 5
--logging_steps 1
--gradient_checkpointing True
--per_device_train_batch_size 4
--gradient_accumulation_steps 1
--deepspeed /Download/swift/deepspeed_config.json
--learning_rate 1e-5
--weight_decay 0.1
--adam_beta2 0.95
deepspeed_config.json:
{
"train_batch_size": 64,
"train_micro_batch_size_per_gpu": 4,
"gradient_accumulation_steps": 1,
"zero_optimization": {
"stage": 2,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
},
"overlap_comm": false,
"contiguous_gradients": false
},
"bf16": {
"enabled": true
},
"gradient_clipping": 1.0,
"wall_clock_breakdown": false,
"steps_per_print": 100
}