Split model between NUMA nodes for CPU inference

[CommanderLake]

If llama.cpp can split a model between GPUs can it split a model between NUMA nodes for CPU inference to eliminate inter-node memory access and take full advantage of the available memory bandwidth?
I ask this because i'm trying to optimize a llama.cpp build for a Xeon Max 9480 and regardless of what i enable in the build, OpenMP, AMX, NUMA etc. there are diminishing returns beyond a single tile worth of threads (28) when in theory if inter-node memory access could be minimised or eliminated by splitting the model evenly and isolating the threads to the correct nodes it could scale far better.

[wkgcass]

You may check this one: #12289
Instead of splitting computing and memory, it would be much easier to copy the model to multiple numa nodes.

The above modification is already implemented and opensourced.

[CommanderLake]

Not if i want to use only the HBM for optimal performance, doing that would limit me to a 16GB model which means no 32B Q4_K_M or above.

[usrlocalben]

Also follow #11733

[CommanderLake]

But how do i use this workaround in my own code using common and llama function calls?
I just tried several methods that might mimic this workaround but it is still not scaling well on more than one tile on the 9480.

[usrlocalben]

And #12088

[CommanderLake]

NUMA and SNC cannot be disabled on the Xeon Max with only HBM.

[CommanderLake]

Using the windows performance counter "NUMA Node Memory\Available MBytes" i can monitor how the memory is allocated and even with the NUMA strategy set to "Distribute" i can clearly see it filling up ONE node and only spilling over to the others when its full.

However, if i use "_putenv("OMP_PROC_BIND=close");" to spread the threads evenly across all NUMA nodes the context or kv cache gets allocated evenly also and its only the model weights that dont get distributed!