Are you frustrated with constant breaking changes?

[jarcen]

I wanted to write a comment in #1026 but maybe I'm alone so I want to know how other people feel about it. (Edit: just as I wrote this the PR removed breaking changes tag and now considers Q4_2)

It seems like there is a nice ecosystem of models forming around this project but it's getting ravaged by repeated file format changes. At least before we had scripts to migrate them to new format but superseding one Q4_0 with another will completely turn quantized weights into pumpkin and no script will fix that, only repeating quantization from source files. That requires huge downloads of original files (>20 Gb if author was charitable to release .bin files in fp16, otherwise double that number). Not to mention pollution of the internet with no longer functional files. Unless you know the 'expiration date' on ggml files introduced by each commit, it's a Russian Roulette for the end user.

I know it's annoying to support growing legacy of formats but at some point the project must turn mature and start handling that.

[slaren]

Considering that all ggml model versions ever released are currently supported by llama.cpp, I don't think that it is very fair to talk about "constant breaking changes".

I know it's annoying to support growing legacy of formats but at some point the project must turn mature and start handling that.

But that's the thing, we are nowhere near that point. If Q4_2 proves to be better than Q4_0 in every meaningful aspect, I would expect that eventually we will remove support for Q4_0. Maintaining old code has a cost and at this point it wouldn't make a lot of sense to slow down development because of that.

If you want to live on the bleeding edge pulling daily from master, breaking changes should be expected. Would it be better if we tagged a 0.1 release and directed people to use that instead?

[jarcen]

You probably saw the edit, this discussion is now less relevant as PR changed to non-breaking.

The change that unified all formats is pretty recent, before that we had to deal with conversion scripts. What's the big deal, you ask? Well, it doesn't synergize with not pulling breaking changes from master. I'm living just like that, with older codebase and had to deal with world moving on and releasing models in newer formats. Some of the changes were justified but some were not. I know it's not your fault and I have no reason to complain when using old codebase but that's the life on the opposite side of "living on a bleeding edge" and it's not pretty too. There comes a time when you have to pull new changes one way or another and pray that your need aligns with a silent week of no surprises.

If it's hard to understand where I'm getting at, look at it another way: there are people who release models with new changes as soon as they appear(or even force you to pull unmerged PRs like Vicuna v1.1, but that's beside the point) forcing you to accept new changes if you want flashy new things. There's also old files that become broken with these changes(even if temporarily). If you want to use both you have to juggle with binaries or wait for a fix. In this rapidly moving field breaking changes are painful even if they are temporary. I simply wish people were more cognizant about making them.

But that's only my voice. As I said, I didn't post a comment there because probably no one cares except me. I just wanted to know what other people think.

[x02Sylvie]

Personally nah.

If something breaks, one can always fetch the previous commit until they got reason to move onto newer one. Or if you're like me then just mish and mash the commits you need

[FNsi]

Nope, and I am expecting for how q4_2 version 65b works.

[PriNova]

A breacking change should always be documented in the commits or best labeled in the PR.
Also, a breacking change should always be the last resort if nothing else works.
Always follow the path of least surprise.

Now that Stable.ai announced to have compatibility with llama.cpp of their new released StableLM in the future, it is much more important to have a structured collaboration of all contributors.