Use FGD to fine-tune the transformer

[cyl943123]

Hi, Cool Work!

I'm curious about the performance of using the FGD to fine-tune the transformer on GLUE task
do you have done it before?

Thanks!!

[belerico]

Hi @cyl943123, nope I haven't tried. I think this will be difficult: even for the simple MNIST a subtle change in the hyperparameters, the learning rate for example, led to instabilities. It will still be nice to see generalization to other tasks. Have you tried something in this regard?

[inikishev]

Hi, Cool Work!

I'm curious about the performance of using the FGD to fine-tune the transformer on GLUE task do you have done it before?

Thanks!!

people have fine tuned language models with MEZO (https://arxiv.org/abs/2305.17333), which is an even more trimmed version of this because instead of calculating JVP exactly it estimates it via finite difference (I haven't tried it though)