[v0.12.2] Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5 8GB

[b4rtaz]

Model: deepseek_r1_distill_llama_8b_q40
Version: 0.12.2

	Evaluation	Prediction
2 x Raspberry Pi 5 8GB	7.70 tok/s	3.54 tok/s
4 x Raspberry Pi 5 8GB	11.68 tok/s	6.43 tok/s

2 x Raspberry Pi 5 8GB

...
🔶 P  278 ms S   288 kB R   522 kB First
🔶 P  258 ms S   288 kB R   522 kB ,
🔶 P  323 ms S   288 kB R   522 kB  I
🔶 P  275 ms S   288 kB R   522 kB  need
🔶 P  293 ms S   288 kB R   522 kB  to
🔶 P  269 ms S   288 kB R   522 kB  understand
🔶 P  281 ms S   288 kB R   522 kB  what

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 7.70 (129.89 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 3.54 (282.22 ms/tok)
⭕ Network is closed

4 x Raspberry Pi 5 8GB

...
🔶 P  162 ms S   864 kB R  1191 kB The
🔶 P  160 ms S   864 kB R  1191 kB  Multi
🔶 P  157 ms S   864 kB R  1191 kB -
🔶 P  176 ms S   864 kB R  1191 kB Device
🔶 P  130 ms S   864 kB R  1191 kB  In
🔶 P  174 ms S   864 kB R  1191 kB ference
🔶 P  132 ms S   864 kB R  1191 kB  Cluster
🔶 P  172 ms S   864 kB R  1191 kB  (
🔶 P  139 ms S   864 kB R  1191 kB MD
🔶 P  184 ms S   864 kB R  1191 kB IC
🔶 P  162 ms S   864 kB R  1191 kB )
🔶 P  156 ms S   864 kB R  1191 kB  is

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 11.68 (85.63 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 6.43 (155.60 ms/tok)
⭕ Network is closed

[NetOpWibby]

Pretty sweet

[danielfalbo]

@Hitomamacs

[dhenson02]

You should go into detail how you did this across multiple

I'd love to read/watch that

[D-i-t-gh]

Thanks for showing your results. Just started to use dllama and it looks very nice!
I use 8 nodes (mostly older Intel CPUs with 4 or 6 cores, AVX2 capable), which are connected to a 2.5G LAN. I get this

...
🔶 P   60 ms S  2016 kB R  2342 kB  less
🔶 P   56 ms S  2016 kB R  2342 kB ,
🔶 P   57 ms S  2016 kB R  2342 kB  which
🔶 P   56 ms S  2016 kB R  2342 kB  is
🔶 P   56 ms S  2016 kB R  2342 kB
🔶 P   62 ms S  2016 kB R  2342 kB 14
🔶 P   63 ms S  2016 kB R  2342 kB .
🔶 P   56 ms S  2016 kB R  2342 kB  Got
🔶 P   84 ms S  2016 kB R  2342 kB  that
🔶 P   63 ms S  2016 kB R  2342 kB  part
🔶 P   56 ms S  2016 kB R  2342 kB  down
🔶 P   56 ms S  2016 kB R  2342 kB .
🔶 P   56 ms S  2016 kB R  2342 kB  Now
🔶 P   56 ms S  2016 kB R  2342 kB ,

Evaluation
   nBatches: 32
    nTokens: 11
   tokens/s: 33.64 (29.73 ms/tok)
Prediction
    nTokens: 245
   tokens/s: 16.63 (60.13 ms/tok)
⭕ Network is closed

This is the command I used (v0.12.7):

./dllama inference --model models/deepseek_r1_distill_llama_8b_q40/dllama_model_deepseek_r1_distill_llama_8b_q40.m --tokenizer models/deepseek_r1_distill_llama_8b_q40/dllama_tokenizer_deepseek_r1_distill_llama_8b_q40.t --buffer-float-type q80 --nthreads 6 --max-seq-len 4096 --workers  192.168.0.151:9990 192.168.0.152:9990 192.168.0.153:9990 192.168.0.154:9990 192.168.0.155:9990 192.168.0.160:9990 192.168.0.161:9990 --prompt "What is 5 plus 9 minus 3?" --steps 256

[LJ-Hao]

Hello b4rtaz, could you tell me what is diffference between evaluation and prediction