[Transform] Extend set of known Hadamard matrices

[kylesayrs]

Purpose

• Enable the construction of Hadamard matrices for more weight shapes, enabling transforms to be applied to more model architectures

Changes

• Add hadamards.safetensors, which stores a list of known Hadamard matrices from http://www.neilsloane.com/hadamard/index.html
  • sizes = [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256]
  • The file size is 1.37Mb, which increases the size of the compressed-tensors package from 1.1Mb to 2.5Mb. I don't expect this to be an issue (LLM Compressor is 3.1Mb), and I don't think this warrants the extra logic and complexity involved with packing and unpacking the values
• Rename _get_hadK to _fetch_hadamard_divisor
  • Implement _fetch_hadamard_divisor to query and fetch matrices from the hadamards.safetensors file
• Clean up hadamard construction functions, add dtype and device args, remove numpy dependence
• Expose is_pow2 as public function for testing, ect.

Hadamard Fetch Script

download_hadamards.py

from typing import Dict, Tuple

import tqdm
import time
import requests
from bs4 import BeautifulSoup

import torch
from safetensors import safe_open
from safetensors.torch import save_file

FILE_PATH = "hadamards.safetensors"
URL_TEMPLATE = "http://www.neilsloane.com/hadamard/{}.txt"
SLEEP_DURATION = 0.1


## Parse text ##

def get_text_content(url: str, session: requests.Session) -> str:
    response = session.get(url)
    html_content = response.text
    soup = BeautifulSoup(html_content, 'html.parser')
    text_only = soup.get_text(separator=' ', strip=True)

    return text_only

def special_preprocess(text: str, name: str) -> str:
    if name == "had.16.0":
        text = text.replace(
            "Had.16.0\nAutomorphism group has order = 10321920 = 2^15 * 3^2 * 5 * 7",
            ""
        )

    return text

def parse_matrix(text: str, size: int) -> torch.Tensor:
    values = []
    for char in text:
        if char in ("+", "1"):
            values.append(1)
        elif char in ("-", "0"):
            values.append(-1)
        elif char in ("\n",):
            pass
        else:
            print(f"unknown: `{char}`")

    return torch.tensor(values, dtype=torch.int8).reshape((size, size))


## Validate ##


def validate_matrix(matrix: torch.Tensor) -> bool:
    matrix = matrix.to(dtype=torch.int64)  # need more range for larger sizes
    return torch.equal(matrix @ matrix.T / matrix.size(0), torch.eye(matrix.size(0)))

def validate_file(file_path: str):
    with safe_open(file_path, framework="pt", device="cpu") as file:
        for key in tqdm.tqdm(file.keys(), desc="Validating file"):
            matrix = file.get_tensor(key)
            assert validate_matrix(matrix)

    print("File contains valid hadamard matrices!")


## Scrape ##


def scrape_matrices() -> Tuple[Dict[str, torch.Tensor], Dict[str, str]]:
    matrices = {}
    metadata = {}

    with requests.Session() as session:
        with open("names.txt") as file:
            for line in file.readlines():
                name = line.strip()
                size = name.split(".")[1]

                if size in matrices:
                    continue

                text = get_text_content(URL_TEMPLATE.format(name), session)
                text = special_preprocess(text, name)
                matrix = parse_matrix(text, int(size))

                if validate_matrix(matrix):
                    matrices[size] = matrix
                    metadata[size] = name

                    print(f"name: {name}")
                    print(f"size: {size}")
                    print("-------------")

                else:
                    print(f"FAILED: {name}")
                    print("-------------")

                time.sleep(SLEEP_DURATION)

    return matrices, metadata


if __name__ == "__main__":
    matrices, metadata = scrape_matrices()
    save_file(matrices, FILE_PATH, metadata)
    validate_file(FILE_PATH)

names.txt

had.1
had.2
had.4
had.8
had.8.1
had.12
had.16.0
had.16.hed
had.16.syl
had.16.twin
had.16.1
had.16.2
had.16.3
had.16.4
had.20.hall.n
had.20.julin
had.20.pal
had.20.toncheviii
had.20.toncheviv
had.20.will
had.20.1
had.20.2
had.20.3
had.24.pal
had.24.1
had.24.2
had.24.3
had.24.4
had.24.5
had.24.6
had.24.7
had.24.8
had.24.9
had.24.10
had.24.11
had.24.12
had.24.13
had.24.14
had.24.15
had.24.16
had.24.17
had.24.18
had.24.19
had.24.20
had.24.21
had.24.22
had.24.23
had.24.24
had.24.25
had.24.26
had.24.27
had.24.28
had.24.29
had.24.30
had.24.31
had.24.32
had.24.33
had.24.34
had.24.35
had.24.36
had.24.37
had.24.38
had.24.39
had.24.40
had.24.41
had.24.42
had.24.43
had.24.44
had.24.45
had.24.46
had.24.47
had.24.48
had.24.49
had.24.50
had.24.51
had.24.52
had.24.53
had.24.54
had.24.55
had.24.56
had.24.57
had.24.58
had.24.59
had.24.60
had.28.pal2
had.28.will
had.32.pal
had.32.syl
had.32.t1
had.32.t2
had.32.t3
had.32.t4
had.36.pal2
had.36.will
had.40.tpal
had.40.ttoncheviv
had.40.twill
had.44.pal
had.44.will
had.48.pal
had.52.will
had.56.tpal2
had.56.twill
had.60.pal
had.60.pal2
had.60.will
had.64.syl
had.68.pal
had.68.will
had.72.pal
had.76.pal2
had.76.will
had.80.pal
had.84.pal
had.84.pal2
had.84.will
had.88.tpal
had.88.twill
had.92.will
had.96.tpal
had.100.will
had.104.pal
had.108.pal
had.108.pal2
had.108.will
had.112.ttpal2
had.112.ttwill
had.116.will
had.120.tpal
had.120.tpal2
had.120.twill
had.124.pal2
had.124.will
had.128.syl
had.132.pal
had.132.will
had.136.tpal
had.136.twill
had.140.pal
had.144.tpal
had.148.pal2
had.148.will
had.152.pal
had.156.will
had.160.tpal
had.164.pal
had.168.pal
had.172.will
had.176.ttpal
had.176.ttwill
had.180.pal
had.184.twill
had.188.tur
had.192.pal
had.196.pal2
had.200.pal
had.204.pal2
had.208.twill
had.212.pal
had.216.tpal
had.220.pal2
had.224.pal
had.228.pal
had.232.twill
had.236.od
had.240.pal
had.244.will
had.248.twill
had.252.pal
had.256.syl

Testing

• Add new hadamard construction and correctness tests
  • Note that large hadamard matrices are too large to be constructed on the cpu, must be done on the GPU. See [Transform] Construct on GPU, cache on CPU #352

[rahul-tuli]

Great find! instead of checking in and shipping the added safetensors file in our wheel; Could we programmatically download it when needed?

[kylesayrs]

@rahul-tuli What’s the benefit to downloading the safetensors file? I understand the bin files are scary, but this introduces another level of file hosting + local artifact management to keep in mind.

Other libraries such as Quark simply copy and paste the entire matrices into python code and ship with that, so clearly shipping with prebuilt data isn’t too problematic for them. I consider a SSOT safetensors file to be an improvement upon that approach.

[rahul-tuli]

@rahul-tuli What’s the benefit to downloading the safetensors file? I understand the bin files are scary, but this introduces another level of file hosting + local artifact management to keep in mind.

Other libraries such as Quark simply copy and paste the entire matrices into python code and ship with that, so clearly shipping with prebuilt data isn’t too problematic for them. I consider a SSOT safetensors file to be an improvement upon that approach.

There's a few reasons behind my ask:

1. Git is not really designed for binary data, the blob is not visible, each time we update this blob (if needed), we won't be able to see the diff it will just show up as a 2MB blob. (I guess maybe that's why some other repos choose to check them in as python code)

2. It is good practice to keep code and data artifacts separate. I see this safetensors file as a versionable/downloadable artifact.

3. Even though this is a small file, why keep it if it's not needed? It sets a bad precedent

I still think hosting, and subsequently downloading this file on first use is better hygiene.

However if @dsikka and @brian-dellabetta feel this is okay, feel free to ignore my ask, won't be blocking the PR over this!

[brian-dellabetta]

awesome!

[dsikka]

What is the point of storing a large subset of small hadamard sizes that we likely won’t see in most models?

[kylesayrs]

@dsikka Is there a reason not to support as many hadamard matrix sizes as we can?

[dsikka]

@dsikka Is there a reason not to support as many hadamard matrix sizes as we can?

If we dont have a usecase for certain sizes because they dont show up in most models, carrying them around seems like a waste of time and space. When do we expect to use hadamards of size 18?

It would make a lot more sense if we storing sizes we expect to see/have seen in popular models as they have a large runtime to compute.

[kylesayrs]

For clarity, having 18 in the registry means that we can generate any hadamard whose size can be represented as 18*(2^n), so even the small values may have a use if they're not covered by the larger values