[v0.12.3] Llama 3.2 1B Q40 on 4 x Raspberry Pi 5 8GB

[b4rtaz]

Llama 3.2 1B Q40

Model: dllama_model_llama3_2_1b_instruct_q40.m
Version: 0.12.3

	Evaluation	Prediction
4 x Raspberry Pi 5 8GB	68.35 tok/s	29.34 tok/s
2 x Raspberry Pi 5 8GB	44.08 tok/s	19.08 tok/s
1 x Raspberry Pi 5 8GB	28.23 tok/s	11.55 tok/s

4 x Raspberry Pi 5 8GB

...
🔶 P   26 ms S   228 kB R   579 kB  connect
🔶 P   28 ms S   228 kB R   579 kB  devices
🔶 P   27 ms S   228 kB R   579 kB  to
🔶 P   28 ms S   228 kB R   579 kB  the
🔶 P   37 ms S   228 kB R   579 kB  cluster
🔶 P   36 ms S   228 kB R   579 kB :

🔶 P   38 ms S   228 kB R   579 kB ``
🔶 P   36 ms S   228 kB R   579 kB `

🔶 P   37 ms S   228 kB R   579 kB #

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 68.35 (14.63 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 29.34 (34.08 ms/tok)

2 x Raspberry Pi 5 8GB

...
🔶 P   60 ms S    76 kB R   318 kB  Edge
🔶 P   62 ms S    76 kB R   318 kB  devices
🔶 P   52 ms S    76 kB R   318 kB  can
🔶 P   41 ms S    76 kB R   318 kB  be
🔶 P   41 ms S    76 kB R   318 kB  used
🔶 P   56 ms S    76 kB R   318 kB  to
🔶 P   60 ms S    76 kB R   318 kB  perform
🔶 P   60 ms S    76 kB R   318 kB  computations
🔶 P   51 ms S    76 kB R   318 kB  at
🔶 P   41 ms S    76 kB R   318 kB  the
🔶 P   41 ms S    76 kB R   318 kB  edge

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 44.08 (22.68 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 19.08 (52.40 ms/tok)
⭕ Network is closed

1 x Raspberry Pi 5 8GB

...
🔶 P   72 ms S     0 kB R     0 kB ##
🔶 P  102 ms S     0 kB R     0 kB  Step
🔶 P   67 ms S     0 kB R     0 kB  
🔶 P   97 ms S     0 kB R     0 kB 3
🔶 P   78 ms S     0 kB R     0 kB :
🔶 P   77 ms S     0 kB R     0 kB  Determine
🔶 P   98 ms S     0 kB R     0 kB  the
🔶 P   67 ms S     0 kB R     0 kB  method

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 28.23 (35.42 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 11.55 (86.60 ms/tok)