✨ New Features and Improvements
- Much Smaller Binaries: In this release we’ve dropped the installed binary size across all platforms (ex. Linux went from ~96MB → ~14MB).
- sdist installation support: Now
hf-xet
can be compiled using best practices for Python package sdist installation. Thanks @tiran and @szalpal for the original bug reports!
🐛 Bug Fixes (retries, open-files, sdist, smaller binaries)
- More resilient uploads & downloads by adding retries to many error paths through download and upload. Fixes #300 #322 #311
- Optimizations around model compression selection, object serialization.
- Prevent "Too Many Open Files" error by limiting concurrent downloads.
- Build & release updates to support sdist, dbg symbols. Fixes #255 #304
- Code cleanup and refactoring around progress reporting.
What's Changed
- Updating hf-xet version to 1.1.1 by @bpronan in #313
- Switch chunk cache to use async RWlock instead of std::sync mutex. by @hoytak in #306
- Fix windows build failure by @hoytak in #316
- Switch file size / byte tracking variables from usize to u64 by @hoytak in #314
- Added opt-test profile option to hf_xet by @hoytak in #307
- Optimize bg4 prediction by @hoytak in #308
- Chunk Partitioning by @ylow in #299
- Move CAS object serialization to before parallel upload gate. by @hoytak in #315
- Consolidate progress reporting code to one crate by @hoytak in #318
- Fixing the sdist install and adding better project information by @bpronan in #319
- Removed unused git-style progress_reporting crate. by @hoytak in #317
- don't bump num concurrent downloads live by @assafvayner in #326
- SDist fix v2 by @bpronan in #325
- Add incremental upload progress and total bytes to progress tracking. by @hoytak in #323
- Back out streaming upload of xorb. by @hoytak in #327
- Extracting debug symbols from wheels by @bpronan in #330
- add retries on body read errors during download by @assafvayner in #331
- Fix release build issues by @bpronan in #332
- hf-xet 1.1.2 release by @rajatarya in #334
- MacOS dbg symbols zipped in release workflow by @rajatarya in #335
- Update release.yml by @rajatarya in #336
- Update dbg assets to upload zip by @rajatarya in #337
- Removing zip step for dbg assets on windows by @rajatarya in #338
Full Changelog: v1.1.1...v1.1.2