Skip to content

Commit

Permalink
sanity check ranges in freeze.py (#1686)
Browse files Browse the repository at this point in the history
* sanity check ranges in freeze.py

this will catch problems earlier and more clearly.

in my case, it appears that deepspeed zero3 sets layer tensor shapes
to [0], which doesn't play well with automatically inferred ranges.
through a bit of luck, inverting ranges still appears to work correctly.

* simplify chained comparison
  • Loading branch information
josharian authored Jul 5, 2024
1 parent c69b7eb commit b3f680d
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions src/axolotl/utils/freeze.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,9 @@ def _merge_ranges(
processed_ranges = [
(start, end if end is not None else layer_size) for start, end in given_ranges
]
for start, end in processed_ranges:
if start < 0 or end > layer_size > 0 or start >= end:
raise ValueError(f"invalid unfreeze range: start={start}, end={end}")

# No need to merge if there's only one or no ranges
if len(processed_ranges) <= 1:
Expand Down

0 comments on commit b3f680d

Please sign in to comment.