Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix type error in extract_lora.py : SVD only supports float32 #510

Closed
wants to merge 4 commits into from
Closed

Fix type error in extract_lora.py : SVD only supports float32 #510

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3594c5d
fix
goliaro Feb 8, 2025
1d298b6
make more concise
goliaro Feb 8, 2025
88fe702
Merge branch 'main' into fix_svd_type
goliaro Feb 17, 2025
47c7f25
Merge branch 'main' into fix_svd_type
cg123 Feb 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion mergekit/scripts/extract_lora.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,10 @@ def arguments(self) -> Dict[str, Any]:
def execute(self, task_vector: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
if self.transpose:
task_vector = task_vector.T
u, s, vh = torch.linalg.svd(task_vector, full_matrices=False)
# convert task_vector to float32 before SVD, then convert results back to original data type
original_dtype = task_vector.dtype
u, s, vh = torch.linalg.svd(task_vector.to(torch.float32), full_matrices=False)
u, s, vh = u.to(original_dtype), s.to(original_dtype), vh.to(original_dtype)
rank = min(self.max_rank, s.shape[0])
if self.sv_epsilon > 0:
rank = min((s > self.sv_epsilon).sum().item(), rank)
Expand Down
Loading