Remove the unused pragma in vit + handle the final layernorm.

Remove the unused pragma in vit + handle the final layernorm. #2287