JSON format for sft stage

[baiyang4]

Hello! Thanks for your fantastic open-source work! Can you provide some examples on the training data format?

[zhuoyang20]

Hi @baiyang4,

I provide an example file which contains two data points for your reference.
dummy.json

Best,
Zhuoyang

[namingsohard]

Hello! Thanks for your outstanding work! I'd like to know how to fine-tune VILA-U using my own data.

1: When multiple images need to be provided for training (give multiple images and generate texts),
2: or when the task requires the model generate images (give prompt and generate images)

what form should the fine-tuning data take? Can these different types of tasks be trained in a single fine-tuning session?

Looking forward to your reply!