support model cards #176
support model cards #176
Conversation
| from PIL import Image | ||
|
|
||
|
|
||
| def save_model_card( |
There was a problem hiding this comment.
Want to keep it minimal yet sufficient for now.
|
Just tested the local changes on a small run: https://huggingface.co/sayakpaul/cog2b_disney (with #178 merged in) LMK if there's anything else left off to be addressed in this PR. |
Yes, that is correct. Would you rather have me run some complete runs for Cog (2b, 5b)? Completely fine by me if so. |
finetrainers/trainer.py
Outdated
| @@ -988,6 +991,8 @@ def validate(self, step: int, final_validation: bool = False) -> None: | |||
| # TODO: this should be configurable here as well as in validation runs where we call the pipeline that has `fps`. | |||
| export_to_video(artifact_value, filename, fps=15) | |||
| artifact_value = wandb.Video(filename, caption=prompt) | |||
| if accelerator.is_main_process: | |||
| prompts_to_filenames[prompt] = filename | |||
There was a problem hiding this comment.
So just tested it and it seems like here the filename ends up becoming something like /raid/aryan/final-validation-.... This is because we join the output_dir to the filename when saving the video
So, I think we need to make sure we use the filename before it gets prefixed with output_dir
There was a problem hiding this comment.
Hmm not sure about it.
I inspected the output_dir:
ls /raid/.cache/huggingface/sayak/cog2b/cog2b_disney
checkpoint-10 pytorch_lora_weights.safetensors validation-5-0-BW_STYLE-A-black-and-whit.mp4
checkpoint-5 README.md validation-5-1-A-woman-with-long-brown-h.mp4
final-10-0-BW_STYLE-A-black-and-whit.mp4 validation-10-0-BW_STYLE-A-black-and-whit.mp4
final-10-1-A-woman-with-long-brown-h.mp4 validation-10-1-A-woman-with-long-brown-h.mp4Seems alright no?
Also https://huggingface.co/sayakpaul/cog2b_disney seems alright in the sense that the resultant model card looks as expected. But LMK if I am missing something obvious.
There was a problem hiding this comment.
I mean to say that the model card does not render the demo video because of the incorrect path to the video relative to the uploaded repository (the case with your repo as well).
In the created model card, the path shouldn't contain the full path with output_dir (for example, right now it creates something like /raid/aryan/cogvideox-lora/validation-video.mp4). But for the model card, it should just be validation-video.mp4)
There was a problem hiding this comment.
Just fixed in 800a47f.
|
The training seems to work phenomenally, so all changes related to Cog seem good! https://wandb.ai/aryanvs/finetrainers-cogvideox/runs/6925e91q. It seems like I used too high a learning rate because the validation videos collapses a bit in some runs, but definitely seems like the model is learning and there is no quality degradation.
Completely up to you if you'd like to release something for a specific style! I did a 1000 step run for Cakify so maybe you could check with Disney dataset if you'd like. T2V lora for effects like cakify is almost always a miss, so will try to do something better with I2V in trainer when we add it. My experiments are more aimed towards figuring out if SNR loss weighting has been implemented correctly in the original Cog codebase as well as our scripts. Will probably be easier to figure out with full finetuning so will put on hold and prioritize memory optimizations |
|
@a-r-r-o-w thanks for the positive note and testing that! I think I will merge this PR given you have verified it. But before that I will wait for your response to #176 (comment). |
|
Sorry for requesting the review again but just want to be sure. Have tested in 800a47f. |
fps.Trainerto keep it leaner.