Poor OCR performance

[tianzhangwu]

It has almost none OCR ablility.

[tianzhangwu]

A more important question is，do you think that methods based on discrete tokens are inherently incapable of performing OCR?

[zhuoyang20]

Hi @tianzhangwu,

We didn't include heavy OCR data during VILA-U's training, and the image resolution is restricted, resulting in poor OCR performance. In our recent work, NVILA, we use dynamic resolution with image tiling techniques and train on more OCR data, leading to strong OCR capabilities. For discrete tokens, it should still be able to perform OCR, but the performance may be slightly worse than with continuous tokens.

Best,
Zhuoyang