Llama 3.2 1B F32 Benchmarks, 8 x Raspberry Pi 4B 8GB

[githuba9f5404]

Running on this hardware setup, overclocked with a clock speed of 2000 MHZ (standard is 1500 MHZ).

$ sudo nice -n -20 ./dllama inference --model ~/dllama_model_llama3.2-1b-instruct_f32.m --tokenizer models/llama3_2_1b_instruct_q40/dllama_tokenizer_llama3_2_1b_instruct_q40.t --steps 64 --prompt 'The Eiffel Tower is' --buffer-float-type f32 --workers 192.168.60.11:9998 192.168.60.12:9998 192.168.60.13:9998 192.168.60.14:9998 192.168.60.15:9998 192.168.60.16:9998 192.168.60.17:9998
💡 arch: llama
💡 hiddenAct: silu
💡 dim: 2048
💡 hiddenDim: 8192
💡 nLayers: 16
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 8192
💡 nSlices: 8
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
📄 chatEosId: 128009
🕒 ropeCacheSize: 8192 kB
⏩ Loaded 5853448 kB
🔶 G  517 ms I  436 ms T   79 ms S 3327744 kB R   1792 kB The
🔶 G  444 ms I  408 ms T   36 ms S   1792 kB R   1792 kB  E
🔶 G  475 ms I  430 ms T   45 ms S   1792 kB R   1792 kB iff
🔶 G  496 ms I  451 ms T   44 ms S   1792 kB R   1792 kB el
🔶 G  468 ms I  420 ms T   47 ms S   1792 kB R   1792 kB  Tower
🔶 G  483 ms I  439 ms T   44 ms S   1792 kB R   1792 kB  is
🔶 G  481 ms I  431 ms T   48 ms S   1792 kB R   1792 kB  a
🔶 G  451 ms I  409 ms T   39 ms S   1792 kB R   1792 kB  must
🔶 G  469 ms I  424 ms T   43 ms S   1792 kB R   1792 kB -
🔶 G  477 ms I  430 ms T   45 ms S   1792 kB R   1792 kB visit
🔶 G  450 ms I  409 ms T   39 ms S   1792 kB R   1792 kB  destination
🔶 G  471 ms I  428 ms T   41 ms S   1792 kB R   1792 kB  in
🔶 G  445 ms I  413 ms T   29 ms S   1792 kB R   1792 kB  Paris
🔶 G  473 ms I  426 ms T   45 ms S   1792 kB R   1792 kB ,
🔶 G  502 ms I  454 ms T   46 ms S   1792 kB R   1792 kB  France
🔶 G  480 ms I  430 ms T   48 ms S   1792 kB R   1792 kB .
🔶 G  489 ms I  449 ms T   38 ms S   1792 kB R   1792 kB  With
🔶 G  470 ms I  427 ms T   41 ms S   1792 kB R   1792 kB  its
🔶 G  448 ms I  407 ms T   38 ms S   1792 kB R   1792 kB  iron
🔶 G  445 ms I  406 ms T   37 ms S   1792 kB R   1792 kB  l
🔶 G  490 ms I  442 ms T   46 ms S   1792 kB R   1792 kB att
🔶 G  442 ms I  402 ms T   38 ms S   1792 kB R   1792 kB ic
🔶 G  485 ms I  440 ms T   43 ms S   1792 kB R   1792 kB ework
🔶 G  437 ms I  393 ms T   42 ms S   1792 kB R   1792 kB  and
🔶 G  441 ms I  400 ms T   39 ms S   1792 kB R   1792 kB  stunning
🔶 G  457 ms I  416 ms T   38 ms S   1792 kB R   1792 kB  views
🔶 G  439 ms I  399 ms T   38 ms S   1792 kB R   1792 kB  of
🔶 G  451 ms I  415 ms T   34 ms S   1792 kB R   1792 kB  the
🔶 G  440 ms I  398 ms T   40 ms S   1792 kB R   1792 kB  city
🔶 G  445 ms I  406 ms T   37 ms S   1792 kB R   1792 kB ,
🔶 G  436 ms I  398 ms T   36 ms S   1792 kB R   1792 kB  it
🔶 G  455 ms I  414 ms T   39 ms S   1792 kB R   1792 kB 's
🔶 G  435 ms I  401 ms T   32 ms S   1792 kB R   1792 kB  a
🔶 G  437 ms I  398 ms T   37 ms S   1792 kB R   1792 kB  symbol
🔶 G  480 ms I  437 ms T   41 ms S   1792 kB R   1792 kB  of
🔶 G  480 ms I  433 ms T   45 ms S   1792 kB R   1792 kB  French
🔶 G  492 ms I  448 ms T   42 ms S   1792 kB R   1792 kB  culture
🔶 G  477 ms I  433 ms T   41 ms S   1792 kB R   1792 kB  and
🔶 G  490 ms I  448 ms T   40 ms S   1792 kB R   1792 kB  engineering
🔶 G  438 ms I  396 ms T   40 ms S   1792 kB R   1792 kB .
🔶 G  496 ms I  447 ms T   47 ms S   1792 kB R   1792 kB  The
🔶 G  453 ms I  411 ms T   40 ms S   1792 kB R   1792 kB  E
🔶 G  443 ms I  401 ms T   40 ms S   1792 kB R   1792 kB iff
🔶 G  437 ms I  395 ms T   40 ms S   1792 kB R   1792 kB el
🔶 G  471 ms I  421 ms T   47 ms S   1792 kB R   1792 kB  Tower
🔶 G  435 ms I  393 ms T   40 ms S   1792 kB R   1792 kB  is
🔶 G  443 ms I  401 ms T   40 ms S   1792 kB R   1792 kB  a
🔶 G  452 ms I  416 ms T   34 ms S   1792 kB R   1792 kB  must
🔶 G  480 ms I  437 ms T   41 ms S   1792 kB R   1792 kB -
🔶 G  488 ms I  446 ms T   40 ms S   1792 kB R   1792 kB visit
🔶 G  481 ms I  434 ms T   45 ms S   1792 kB R   1792 kB  destination
🔶 G  453 ms I  415 ms T   36 ms S   1792 kB R   1792 kB  in
🔶 G  439 ms I  394 ms T   43 ms S   1792 kB R   1792 kB  Paris
🔶 G  446 ms I  399 ms T   45 ms S   1792 kB R   1792 kB ,
🔶 G  468 ms I  417 ms T   49 ms S   1792 kB R   1792 kB  France
🔶 G  478 ms I  430 ms T   46 ms S   1792 kB R   1792 kB .
🔶 G  478 ms I  427 ms T   49 ms S   1792 kB R   1792 kB  With
🔶 G  457 ms I  410 ms T   44 ms S   1792 kB R   1792 kB  its
🔶 G  467 ms I  420 ms T   45 ms S   1792 kB R   1792 kB  iron
🔶 G  436 ms I  397 ms T   37 ms S   1792 kB R   1792 kB  l
🔶 G  474 ms I  424 ms T   48 ms S   1792 kB R   1792 kB att
🔶 G  441 ms I  400 ms T   39 ms S   1792 kB R   1792 kB ic
🔶 G  452 ms I  397 ms T   53 ms S   1792 kB R   1792 kB ework
🔶 G  451 ms I  411 ms T   38 ms S   1792 kB R   1792 kB  and
Generated tokens:    64
Avg tokens / second: 2.16
Avg generation time: 462.50 ms
Avg inference time:  418.55 ms
Avg transfer time:   41.97 ms