Fine-tuning from an pre-trained image checkpoint has mismatch issue

[Amshaker]

Hi @xiaoqian-shen ,

I managed to do the image pretraining as you explained in the readme, but could you please clarify whether I should change image_token_len and query_num_list in config.json from 576 to 144 in which config.json exactly for the video training stage?

I have three config.json:
(1) config.json of the the LLM itself.
(2) config.json of the output of the image training stage "--output_model_filename" (checkpoints/cambrian_qwen)
(3) config.json of the saved checkpoint from the image training stage (checkpoints/cambrian_qwen/checkpoint-14318)

When I tried to change the config.json of the saved model "--output_model_filename" (checkpoints/cambrian_qwen), I couldn't load the pre-trained model due to a size mismatch in the vision_sampler between the saved image checkpoint (with image_token_len 576) and the current model (with image_token_len 144).

Shall I ignore this mismatch??