Replies: 9 comments 25 replies
-
|
@ubergarm may just want to note in the math that Bpw is converted to bytes from bits with your constant term (/8). Initially confused me. Otherwise great write up! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! I added a note to make it more clear and made it simply |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm great work you have done! I am living in it seems that the intel amx team are implementing fp8 support. if so, the hardware is really great for I am curious about the concurrent performance. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I wonder how much they cost compared to the MSRP sticker cost? The best current performance would come from ktransformers with a one or two 4090D depending on how much context you need. You need enough RAM to hold the entire model twice if you go with a dual socket system. I have benchmarked the best I could get with llama.cpp main branch today. Hopefully there are some easy optimizations that I missed. If the Intel AMX team implements fp8 right on the CPU that could get interesting for sure! 谢谢，我想知道它们的价格与厂商建议零售价相比如何？ 目前最佳性能来自配备一个或两个4090D的ktransformers，具体取决于您所需的上下文长度。如果采用双插槽系统，您需要足够的内存来容纳整个模型两次。 目前我已经对llama.cpp主分支进行了基准测试，得到了目前我能获得的最佳结果。希望还有一些我遗漏的简单优化方法。如果英特尔AMX团队能在CPU上正确实现浮点8精度，那肯定会非常有趣！ |
Beta Was this translation helpful? Give feedback.
-
now |
Beta Was this translation helpful? Give feedback.
-
|
有趣的是，CPU与GPU的指令集及内存支持技术似乎正呈现融合趋势。英特尔Diamond Rapids平台引入 interesting how there seems to be a convergence between CPU and GPU instruction sets and memory support. Intel Diamond Rapids with |
Beta Was this translation helpful? Give feedback.
-
|
Nice writeup, I too have access to a similar system for a bit. I did not have much luck getting deepseekR1 to convert to gguf or I would just hand you my numbers. However, have you tried building llama.cpp with oneAPI Intel compiler? It will use oneMKL as the BLAS backend. Steps Are:
source /opt/intel/oneapi/setvars.sh
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_NATIVE=ON -DCMAKE_INSTALL_PREFIX=~/llama_build/ -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_SERVER=ON GGML_CCACHE=OFF
cmake --build build --config Release
|
Beta Was this translation helpful? Give feedback.
-
|
Great this is the kind of thing I figured existed somewhere, but I'm not super familiar with Intel ecosystem. I'll try this out and see if it helps! UPDATE: I tried it an no gains and pp was much slower oddly. I show the instructions as well as the exact benchmark command if possibly there is something I missed. I only tested a single NUMA node hoping to minimize variability with NUMA stuff. The Intel Base Kit Compiler may actually be faster on some workloads but this particular one is likely memory bandwidth bottle-necked and not CPU performance. |
Beta Was this translation helpful? Give feedback.
-
|
Can you guys share how to compile AMX on Windows? |
Beta Was this translation helpful? Give feedback.
-
|
I'm also going through these discussions and had the same results. OpenMP (the default cpu backend) performed exactly the same as the intel one. The bottleneck is not on the CPU (even though they go up to 100% when running) |
Beta Was this translation helpful? Give feedback.
-
|
Some time ago I started similar discussion with numbers for AMD Epyc: #11733 I identified several areas where performance is reduced because of NUMA and got discouraged by the effort needed to straighten this up (I guess I'm not the first one). |
Beta Was this translation helpful? Give feedback.
-
|
I updated the original post with links to your amazing thread with interesting hacks like warming up with token generation first. |
Beta Was this translation helpful? Give feedback.
-
|
I tried having multiple CPU rpc-server backends in different NUMA nodes, but seems like they default to 4 threads |
Beta Was this translation helpful? Give feedback.
-
I guess it could not help a lot, the existing splitting strategy is by layer, which could not speed things up on single request. may be on batch mode. |
Beta Was this translation helpful? Give feedback.
-
|
I have one more testing loading right now with Nope, no 🎲 ... Still really slow unfortunately. Updated notes in #11397 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
I've experienced similar results running Deepseek R1 (unsloth 2.22b). Regarding the comment about SMT being faster than having it disabled, this was a property of intel cpus and the opposite happened with AMD. For details on this check the paper "Placement of Virtual Containers on NUMA systems: A Practical and Comprehensive Model" Now... I'd love to be able to have both numa nodes working, since using them both produces less tokens/s than using a single node with half the processes. Right now I found the best performance disabling ALL cuda devices, using numactl to use a single node, turning off the cpu security mitigations, setting the k cache type to fp16, disabling autobalancing,(still didn´t try adding interleave). On my hardware I'm getting around 2.3 tokens/s. Today I've been having to also disable mmap, otherwise the IO halts the entire process (this wasn't the case yesterday. I need to investigate). My hardware:
|
Beta Was this translation helpful? Give feedback.
-
|
What is the output of I assume it is a total of 4x NUMA nodes across 2x sockets? That would mean each node has I agree memory bandwidth is key. It would be nice if there were a way to take advantage of the aggregate bandwidth across all NUMA nodes for single user generation situation. The only viable solution of which I know today is to use AMD Epyc with BIOS set to NPS0. On the larger dual socket 6980P there are 6x total NUMA nodes, each has only 256GB in my 1.5TB RAM setup so I can't even fit that quant into a single node haha... We'll see how the tensor parallel, parallel pipeline, and specialized distributed MoE "data parallel" features pan out across the different LLM inference engines. |
Beta Was this translation helpful? Give feedback.
-
|
I just learned about Intel Xeon Given none of the inference engines seem optimized for multiple NUMA right now, I want to try to get my test system BIOS set to At least the next couple months will probably be the best option for now in combination with loading the entire model weights into RAM twice (once per socket). There is a very experimental proposal for it here Eventually Tensor Parallel / Data Parallel + MoE Parallel / Pipeline Parallel type implementations might allow for better performance optimizing across more smaller NUMA nodes. Maybe you and everyone already knows about this and had it disabled already, but I just learned about it! 😅 |
Beta Was this translation helpful? Give feedback.
-
|
that's too new for my aging system xD |
Beta Was this translation helpful? Give feedback.
-
heart broken, in the |
Beta Was this translation helpful? Give feedback.
-
|
@ice6 |
Beta Was this translation helpful? Give feedback.
-
|
hello: |
Beta Was this translation helpful? Give feedback.
-
Thanks for sharing! Look like your best speed on the unsloth A couple questions:
Thanks! 感谢分享！您在unsloth
感谢！ |
Beta Was this translation helpful? Give feedback.
-
|
1.numactl --interleave=all \ , and I also test disabled NUMA in BIOS but benchmark is a little bit lower than above setting . ( 6token/s) 2.Meituan INT8 model seems not compatible to popular Tool like llama.cpp and Vllm, just limited to SLANG; well ,i also try to connect with meituan in other approches.( like Chinese Model site and Wechat) But still no responese currenty.
|
Beta Was this translation helpful? Give feedback.
-
Update my combination test: Dual CPU Socket for Q4_k_m
BTW, how to disable one CPU in BIOS? just like your Single Socket Test. |
Beta Was this translation helpful? Give feedback.
-
|
maybe |
Beta Was this translation helpful? Give feedback.
-
Yeah, I believe that llama.cpp does make use of However, I don't believe it makes use of
I only know for Intel Xeon 6th generation. Go into BIOS and set I updated my benchmarks above and setting 1x "big" NUMA node per CPU is faster than Thanks for sharing info! |
Beta Was this translation helpful? Give feedback.
-
|
There are some OS kernel tweaks that might be worth trying:
|
Beta Was this translation helpful? Give feedback.
-
|
AMX tile config is here in llama.cpp If the tensor OP type is GGML_OP_MUL_MAT, it will be invoked on Intel AMX supported platform. |
Beta Was this translation helpful? Give feedback.
-
|
hello:
what's more, llama.cpp seems more sophisticated in DRAM setting, it's prefere using Dram Cache instead of direct DRAM injecting. |
Beta Was this translation helpful? Give feedback.
-
Intel Xeon performance on R1 671B quants?
Last Updated On: Tue Mar 18 12:11:53 AM EDT 2025
tl;dr;
UPDATE: Fastest CPU only benchmarks to date are with FlashMLA-2 and other optimizations on ik_llama.cpp fork.
UPDATE: Interesting post regarding AMX optimizations and DeepSeek-R1.
UPDATE: Definitely check out @fairydreaming's deep dive on similar issues here
llama.cpp seems to run best with all memory in a single NUMA Node as of Q1 2025.
So configure in BIOS a single NUMA node per CPU socket and only use a single CPU socket e.g.
SNC=Disableon 6th generation Intel Xeon.If you have AMD Epyc that supports
NPS0, use that for best performance.Eventually if you have enough RAM to hold the entire model twice, you can use data parallel to load weights duplicated for each CPU socket single NUMA node (see ktransformers for that, and possibly llama.cpp experimental branch coming).
Otherwise, using multiple CPU sockets potentially degrades performance on single inference workloads due to cross NUMA access latency and bandwidth bottleneck.
Overview
I have limited access to some fairly new high end Intel Xeon servers including a dual socket 6980P Level1Techs YouTube 6980P Review and hopefully soon the recently available 6787P Level1Techs YouTube 6787P Review.
As there is no GPU installed on this specific dual 6980P rig, I am skipping testing ktransformers for now, and testing llama.cpp CPU only inferencing of various R1 671B GGUF quants.
I'm curious if others have any tips on how to improve performance for similar configurations specifically newer Intel Xeon CPUs with AMX extensions in dual or single socket configurations.
Especially how is the best way to take advantage of both CPU sockets simultaneously?
6980P Benchmarks
Here are the high level results of my initial llama-bench testing for token generation. Methodology details and discussion provided below.
Default BIOS
SNC=Auto/Enablefor 3x NUMA Nodes per CPU SocketAfter setting BIOS
SNC=Disable, basically same as AMD Epyc'sNPS1, 1x NUMA Node per CPU Socketllama.cpp@?ba765438?ik_llama.cpp@f2fb15dellama.cpp@?ba765438?llama.cpp@?ba765438?ik_llama.cpp@f2fb15dellama.cpp@?ba765438?ik_llama.cpp@f2fb15deRelated Issues
Potentially related issues include:
ktransformers
The ktransformers project is doing some interesting things specific to Intel Xeon optimizations:
USE_NUMA=1flag seems to copy the entire model weights into memory twice (once for each CPU socket?) presumably to alleviate cross socket UPI link bottlenecks?Theory
Assuming memory bandwidth is the limiting factor and not CPU bottleneck, the theoretical maximum token generation speed can be calculated with:
Formula
Definitions
Example Calculation
For 225 GB/s aggregate RAM bandwidth running Q2@2.51 bits-per-weight quantization:
Discussion
Some thoughts, musings, and wild speculations:
SNC=Disableon newer Intel Xeon 6th Generation CPUs for 1x NUMA Node per CPU socket.-ot exps=CPUstuff for NUMA nodes? lol...numactl --interleave=0,1,2and--numa numactlgives only 1.3x better performance over single node.SNC=Disableon newer Intel Xeon 6th Generation CPUs for 1x NUMA Node per CPU socket.send()calls it is very very slow.load_tensors: tensor 'token_embd.weight' (q8_0) (and 54 others) cannot be used with preferred buffer type AMX, using CPU insteadamx_tileamd_int8may benefit fromint8block-wise quant.Conclusion
A few things learned along the way:
--numa distributedoes not seem to honornumactlnortasksetprocessor affinity and you may end up with CPU cores on nodes other than memory nodes. So you might want to use--numa numactland double check withnumastatandbtopto make sure cores and memory allocating how you expect.-mmap 0to pay attention which NUMA nodes get used as it might not distribute evenly or optimally.tg 5.43 ± 0.02 @ 108 threadswith default CPU build vstg 5.63 ± 0.02 @ 108 threadswith AMX explicitly enabled as shown above.Cheers and thanks for your time! Good luck to everyone in the quest for more tok/sec!
Methodology and Notes
Click the arrow to open the fold filled with benchmarking logs and notes.
Methodology and Notes
Methodology and Notes
System Information
Memory Benchmarks
Compile AMX Extensions
Single NUMA Node
UD-Q2_K_XLBenchmarking Stock Compiler$ numactl -N 0 -b -m 0 -C 0-42,256-298 \ ./build_amx/bin/llama-bench \ --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \ --cache-type-k f16 \ --cache-type-v f16 \ --numa numactl \ --threads 43,16,32,43,54,86 $ watch sudo numastat -p $(pidof llama-bench) Per-node process memory usage (in MBs) for PID 3113697 (llama-bench) Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Total --------------- --------------- --------------- --------------- --------------- --------------- --------------- Huge 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Heap 38.61 0.00 0.00 0.00 0.00 0.00 38.61 Stack 0.03 0.00 0.00 0.00 0.00 0.00 0.03 Private 223061.28 0.00 0.00 1.72 0.14 0.00 223063.14 ---------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- Total 223099.92 0.00 0.00 1.72 0.14 0.00 223101.78Results
build: a800ae46 (4783)Single NUMA Node
UD-Q2_K_XLBenchmarking Intel Base Kit CompilerNo improvements here and pp512 regressions over compiling the AMX way or even defaults. Likely indicates a memory bandwidth bottle neck and not CPU performance...? Need flame graphs lol...
$ numactl -N 0 -b -m 0 -C 0-42,256-298 \ ./build_intel/bin/llama-bench \ --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \ --cache-type-k f16 \ --cache-type-v f16 \ --numa numactl \ --threads 43,16,32,43,54,86 $ watch sudo numastat -p $(pidof llama-bench) Per-node process memory usage (in MBs) for PID 3266837 (llama-bench) Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Total --------------- --------------- --------------- --------------- --------------- --------------- --------------- Huge 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Heap 63.23 0.00 0.00 0.00 0.00 0.00 63.23 Stack 0.09 0.00 0.00 0.00 0.00 0.00 0.09 Private 223392.67 0.00 0.00 1.43 0.06 0.00 223394.16 ---------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- Total 223455.99 0.00 0.00 1.43 0.06 0.00 223457.48Results
build: a800ae46 (4783)Single Socket
UD-Q2_K_XLBenchmarkingResults
NOTE: I deleted the first pp/tg warm-up runs which used 128 threads to distribute memory across nodes.
build: a800ae46 (4783)(more threads is worse in other similar benchmarks I've run so I didn't search that space)
NOTE: This screenshot was before I switched from
--numa distributeto--numa numactland you can see CPU cores active on the opposite socket as where memory was allocated causing even worse performance.Single Socket
Q4_K_MBenchmarking$ numactl -N 0,1,2 -b -m 0,1,2 -C 0-127,256-383 \ ./build_amx/bin/llama-bench \ --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/DeepSeek-R1-Q4_K_M-00001-of-00009.gguf \ --cache-type-k f16 \ --cache-type-v f16 \ --numa numactl \ --threads 128,64,86,108,128 $ watch -n1 numastat -p $(pidof llama-bench) Per-node process memory usage (in MBs) for PID 3088631 (llama-bench) Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Total --------------- --------------- --------------- --------------- --------------- --------------- --------------- Huge 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Heap 28.45 0.27 10.02 0.00 0.00 0.00 38.73 Stack 0.05 0.00 0.00 0.00 0.00 0.00 0.05 Private 128077.27 127615.25 134876.96 1.72 0.14 0.00 390571.34 ---------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- Total 128105.76 127615.51 134886.98 1.72 0.14 0.00 390610.12Results
build: a800ae46 (4783)Dual Socket
Q4_K_MBenchmarking Take 1Take 2 below turned out slightly faster.
$ numactl -N 0-5 -b -m 0-5 -C 0-511 \ ./build_amx/bin/llama-bench \ --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/DeepSeek-R1-Q4_K_M-00001-of-00009.gguf \ --cache-type-k f16 \ --cache-type-v f16 \ --numa numactl \ --threads 256,86,128,172,216,256,296 $ watch -n1 numastat -p $(pidof llama-bench) Per-node process memory usage (in MBs) for PID 3107662 (llama-bench) Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Total --------------- --------------- --------------- --------------- --------------- --------------- --------------- Huge 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Heap 0.05 9.89 0.40 0.04 28.47 0.02 38.87 Stack 0.00 0.00 0.02 0.00 0.06 0.00 0.08 Private 62916.21 65258.70 64395.89 64235.00 64886.67 64430.31 386122.79 ---------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- Total 62916.26 65268.59 64396.31 64235.04 64915.20 64430.34 386161.73Results
build: a800ae46 (4783)Dual Socket
Q4_K_MBenchmarking Take 2This turned out slightly faster than Take 1 above.
Results
Single Socket
Q8_0Benchmarking$ numactl -N 0,1,2 -b -m 0,1,2 -C 0-127,256-383 \ ./build_amx/bin/llama-bench \ --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q8_0/DeepSeek-R1.Q8_0-00001-of-00015.gguf \ --cache-type-k f16 \ --cache-type-v f16 \ --numa numactl \ --threads 128,64,86,108,128 $ watch -n1 numastat -p $(pidof llama-bench) Per-node process memory usage (in MBs) for PID 3091581 (llama-bench) Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Total --------------- --------------- --------------- --------------- --------------- --------------- --------------- Huge 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Heap 31.20 7.51 0.00 0.00 0.00 0.00 38.71 Stack 0.03 0.02 0.00 0.00 0.00 0.00 0.05 Private 235043.34 229996.21 228409.38 1.72 0.14 0.00 693450.79 ---------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- Total 235074.57 230003.74 228409.38 1.72 0.14 0.00 693489.55Results
build: a800ae46 (4783)Dual Socket
Q8_0Benchmarking$ numactl -N 0-5 -b -m 0-5 -C 0-511 \ ./build_amx/bin/llama-bench \ --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q8_0/DeepSeek-R1.Q8_0-00001-of-00015.gguf \ --cache-type-k f16 \ --cache-type-v f16 \ --numa numactl \ --threads 256,128,172,216,256,296 $ watch -n1 numastat -p $(pidof llama-bench) Per-node process memory usage (in MBs) for PID 3101638 (llama-bench) Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Total --------------- --------------- --------------- --------------- --------------- --------------- --------------- Huge 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Heap 0.02 2.00 0.03 2.03 34.78 0.00 38.86 Stack 0.00 0.00 0.00 0.01 0.07 0.00 0.08 Private 109025.81 108656.96 109494.54 121461.80 119104.16 120311.99 688055.26 ---------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- Total 109025.83 108658.96 109494.57 121463.85 119139.00 120311.99 688094.20Results
build: a800ae46 (4783)Intel oneAPI Base Toolkit
RPC Experiment Take 1
I managed to use the experimental llama.cpp RPC backend feature. Essentially I started up a process on one CPU socket with 200GB RAM available. Then I started llama-server on the other CPU socket specifying the RPC endpoint as a
--deviceso it acts like a GPU. Then you "offload" layers to the RPC device. This was nice in that I could keep all processing local to a single NUMA node on each CPU socket. However, the performance was much worse likely due to synchronous stages stalling and barely utilizing CPU cores as described in this github issue comment. It got maybe 2 tok/sec after fully warmed up... :oof:UPDATE I realized the rpc-server uses
GGML_DEFAULT_N_THREADSwhich was set to 4. Write and test patch to force CPU backend and specify threads.UPDATE 2 I made the patches and tried more stuff over in this github issue comment but still no better...
RPC Experiment Take 2
Trying to keep MoE block processing and memory in single NUMA node, but this experiment got only 2 tok/sec.
RPC Experiment Take 3
SNC=DisableModeThis will give us 1x NUMA Node per CPU Socket.
All following tests done on
build: e0331326 (4696)which is essentiallysl/custom-tensor-offloadandug/rpc-numa-cpu-backendcompiled with RPC enabled, but not actually using any of those features.Memory Benchmarks
1x CPU
UD-Q2_K_XLResults
ik_llama.cpp
Results
2x CPU
UD-Q2_K_XLResults
1x CPU
Q4_K_MResults
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------------: | -------------------: |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 64 | pp512 | 24.40 ± 4.96 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 64 | tg128 | 6.95 ± 0.33 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 43 | pp512 | 25.41 ± 0.70 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 43 | tg128 | 5.88 ± 0.04 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 64 | pp512 | 33.83 ± 0.50 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 64 | tg128 | 7.30 ± 0.02 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 86 | pp512 | 39.24 ± 0.94 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 86 | tg128 | 7.82 ± 0.02 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 128 | pp512 | 46.84 ± 1.46 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | RPC | 99 | 128 | tg128 | 7.16 ± 0.02 |
ik_llama.cpp
Results
2x CPU
Q4_K_MResults
TODO
1x CPU
Q8_0Results
ik_llama.cpp
Results
2x CPU
Q8_0Results
TODO
Beta Was this translation helpful? Give feedback.
All reactions