Add Karcher Mean Merge Method

[win10ogod]

Add Karcher Mean Merge Method

Description

This PR adds a new merge method based on the Riemannian (Karcher) mean concept. The Karcher mean provides a geometrically meaningful way to average points on a manifold, which is particularly useful for merging model weights that can be interpreted as points on a hypersphere.

Features

• Implements the Karcher mean algorithm for weight fusion
• Supports configurable parameters:
  • max_iter: Maximum iterations for the Karcher mean algorithm (default: 10)
  • tol: Convergence tolerance (default: 1e-5)
• Includes comprehensive tests for various scenarios

Implementation Details

• The implementation follows the existing merge method patterns in mergekit
• Properly handles tensor normalization and scaling
• Includes detailed documentation and type hints
• Adds tests for basic functionality, custom parameters, multi-model merging, and VLM compatibility

References

• Karcher Mean on Wikipedia
• my github repository

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[6DammK9]

Warning: abstract math

I think this may be coincident with the "Geometric Median" in #345, which has been included in a merged PR in other repo.
Mathmatically (well, I'm not in Math major), Geometric median would be an Euclidean representation of Karcher Mean, which is also the Riemannian Center of Mass under Riemannian manifold. The iterlative Weiszfeld's algorithm highly resembles the proposed algorithm, although I'm not sure if the "trigonometry functions with norm" is actually equivalent to L2 norm.

Anyway nice work!

[win10ogod]

Warning: abstract math

I think this may be coincident with the "Geometric Median" in #345, which has been included in a merged PR in other repo. Mathmatically (well, I'm not in Math major), Geometric median would be an Euclidean representation of Karcher Mean, which is also the Riemannian Center of Mass under Riemannian manifold. The iterlative Weiszfeld's algorithm highly resembles the proposed algorithm, although I'm not sure if the "trigonometry functions with norm" is actually equivalent to L2 norm.

Anyway nice work!

"Slightly different", this algorithm assumes that the model's intrinsic space is nonlinear.
And Karcher mean iteration was used.

[win10ogod]

This is my recent work, and there is no paper yet.
I hope you can please merge the PR submission @cg123

[cg123]

Thanks for the PR, this is quite interesting! I'd love to merge it in. Could you run the pre-commit hook to get the formatting standardized?

[win10ogod]

Thanks for the PR, this is quite interesting! I'd love to merge it in. Could you run the pre-commit hook to get the formatting standardized?

Could you please guide me? This is my first time submitting a PR.

[cg123]

First make sure you have the dev dependencies installed, like so:

pip install -e .[dev]

And then run this command:

pre-commit run --all-files

That will autoformat the code, then you can add those changes and push them.

[win10ogod]

1 workflow awaiting approval

What should I do when I encounter 1 workflow awaiting approval?
Just need to wait?

[win10ogod]

1 workflow awaiting approval

[win10ogod]

@cg123 Could you take a look? It should work now.

[cg123]

Thanks for the PR! Merged.