Minimum gpus for pretraining Yi-34

[chuan298]

Hi team and @winglian ,
I get OOM error when pretraining Yi-34 with 8*A100 80GB, flash_attn, deepspeed zero3, sequence_len=2048. So what is minimum gpus for this?

[NanoCode012]

Hey, can you try enable cpu offload and reduce batch size? Theoretically, it should fit.