Add Cosmopolitan libc for cross-platform binaries (like llamafile)

[BradHutchings]

Hi all,

I use and ship llamafile builds. Find them here:

https://huggingface.co/bradhutchings/Brads-LLMs

I am a huge fan of what the llamafile project has done, as should be obvious. I've tried to help out with the effort. For example, granite support. However... I need llamafile for cross-platform binaries, not for their take on what llama.cpp should be. I'm very happy with the plain llama.cpp server. I ship llamafile with my own completion (not chat) focused UI. I need support for latest models, like Gemma 3, quickly, not weeks or months after llama.cpp has it.

So, would anyone who has solid cmake skills like to help me adapt llama.cpp with an option to build with Cosmo? At least the core stuff and server example to start. I'd like to do it in a way that doesn't get in the way of the current build system. Ideally, it would be a flag for cmake or something to uncomment out in a configuration file.

I have had a little feature incorporated into llama.cpp -- the model loading screen in the server. So I'm not a newbie to this project. I just don't know where to start with integrating Cosmo and making it palatable for this project.

And I have no idea about the politics of these two projects. It seems like llamafile kinda spun off their own direction, and that was cool for awhile, but it's a PITA now.

Anyone interested?

-Brad

[BradHutchings]

Zero interest in this. Wow!!

Well, I have made some progress. Here's an outline:

1. After a bunch of feasibility research and fiddling, I decided to resurrect and update the old Makefile rather than try to make cosmocc work with CMake. The latter is obviously (?) the best approach, especially for future compatibility with the llama.cpp project, but is beyond my ability to figure out.
2. I got the Makefile to build all targets with native tools on Ubuntu 24.04, so that llama-server would run locally on the build machine.
3. I was able to build all targets with cosmocc and suppress most warnings. I have one outstanding problem, which I will get into later.
4. I was able to run llama-server built with cosmocc on my Ubuntu x86_64 build virtual machine. I was also able to run the same binary on a Windows 11 x86_64 laptop, and load the latest Google Gemma 3 4B-Instruct.
5. I hope to test whether the binary runs on an M3 or M4 Mac Mini soon.
6. I plan to test whether the binary runs on a Raspberry Pi 5.

At this time, I'm not sure whether GPU support gets build correctly, but I am very happy just to have CPU inference as a first step.

Here's a picture from the Windows Terminal app starting it up:

And here's a picture of the UI in the browser, having replied to my first query:

So here is the only real build problem I ran into. Cosmo's library doesn't seem to have std::fill in it. So I had to comment out a couple lines in llama-context.cpp to get the whole thing to compile. Is there a C++ expert here who could write me some inline code to fix that? In parallel, I'll ask the Cosmopolitan project how to get that in their library.

---

So now, is anyone interested in this? The point is to have something like llamafile, but up to date with latest llama.cpp. Anyone????

[BradHutchings]

Gemma 3 1B Instruct running on a Pi 5 with a cosmocc compiled llama-sever and my Completion Tool front end. This has a couple glitches I can fix, but if you need a household LLM appliance, holy smokes! About 10 tokens/sec, pretty good prose!