[v0.12.3] Llama 3.2 3B Q40 on 4 x Raspberry Pi 5 8GB

[b4rtaz]

Llama 3.2 3B Q40

Model: dllama_model_llama3_2_3b_instruct_q40.m
Version: 0.12.3

	Evaluation	Prediction
4 x Raspberry Pi 5 8GB	27.74 tok/s	13.68 tok/s
2 x Raspberry Pi 5 8GB	23.63 tok/s	10.33 tok/s
1 x Raspberry Pi 5 8GB	14.30 tok/s	5.95 tok/s

4 x Raspberry Pi 5 8GB

...
🔶 P   64 ms S   571 kB R   911 kB import
🔶 P   59 ms S   571 kB R   911 kB  torch
🔶 P   83 ms S   571 kB R   911 kB 

🔶 P   64 ms S   571 kB R   911 kB from
🔶 P   59 ms S   571 kB R   911 kB  transformers
🔶 P   83 ms S   571 kB R   911 kB  import
🔶 P   84 ms S   571 kB R   911 kB  Auto

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 27.74 (36.05 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 13.68 (73.08 ms/tok)

2 x Raspberry Pi 5 8GB

...
🔶 P   97 ms S   190 kB R   429 kB  IoT
🔶 P   97 ms S   190 kB R   429 kB  infrastructure
🔶 P   97 ms S   190 kB R   429 kB ,
🔶 P   97 ms S   190 kB R   429 kB  allowing
🔶 P   97 ms S   190 kB R   429 kB  for
🔶 P   97 ms S   190 kB R   429 kB  more
🔶 P   97 ms S   190 kB R   429 kB  efficient
🔶 P   97 ms S   190 kB R   429 kB  use

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 23.63 (42.32 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 10.33 (96.79 ms/tok)
⭕ Network is closed

1 x Raspberry Pi 5 8GB

🔶 P  168 ms S     0 kB R     0 kB  a
🔶 P  169 ms S     0 kB R     0 kB  powerful
🔶 P  168 ms S     0 kB R     0 kB  cluster
🔶 P  169 ms S     0 kB R     0 kB .


🔶 P  168 ms S     0 kB R     0 kB ##

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 14.30 (69.95 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 5.95 (168.06 ms/tok)