Distributed Inference Test with 8x Qualcomm Snapdragon 8 Gen 2 Nodes

[alphadance]

Configuration:

• 8-node cluster using Qualcomm Snapdragon 8 Gen 2 processors
• Network initialization completed across all nodes (10.32.xx.xx)
• Model Architecture: Llama
• Context Window: 131K tokens
• Key parameters:

  HiddenDim: 14336
  nLayers: 32
  nHeads: 32 (8 key-value heads)
  RopeScaling: f=8.0, l=1.0, h=4.0, o=8192
  

Performance Results:

Evaluation
   nBatches: 32
    nTokens: 6
   tokens/s: 6.67 (149.83 ms/tok)
Prediction
    nTokens: 10
   tokens/s: 5.08 (197.00 ms/tok)

Full Log Output:

Click to expand

[Model Config]
📄 BosId: 128000
📄 EosId: 128001
📄 RegularVocabSize: 128000
📄 SpecialVocabSize: 256
kvDim=1024, nNodes=8 
💡 Arch: Llama
💡 HiddenAct: Silu
💡 Dim: 4096
💡 HiddenDim: 14336
💡 VocabSize: 128256
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 8
💡 SeqLen: 131072
💡 NormEpsilon: 0.000010
💡 RopeType: Llama3.1
💡 RopeTheta: 500000
💡 RopeScaling: f=8.0, l=1.0, h=4.0, o=8192

[Hardware]
📀 RequiredMemory: 7048367 kB
🧠 CPU: neon fp16

[Network Status]
⭕ All 8 nodes connected successfully:
- 10.32.59.249:50052
- 10.32.59.235:50052
- 10.32.59.253:50052
- 10.32.59.242:50052
- 10.32.49.204:50052
- 10.32.49.220:50052
- 10.32.49.196:50052

[Inference Test]
Prompt: "what is 3+5"
🔷 Initial Evaluation: 899 ms
🔶 Token generation latency: 160-359 ms/token
Final Response: "divided by 2 times 4? That is"

---

Unexpected Observation ❓

Despite using 8x Snapdragon 8 Gen 2 nodes (theoretically more powerful than Raspberry Pi clusters), the performance metrics failed to surpass those achieved by 4x Raspberry Pi nodes in previous tests.

Key questions:

1. What might be causing this performance bottleneck?
2. Could this be related to:
   • Network synchronization overhead?
   • Memory bandwidth limitations?
   • Suboptimal model partitioning?
3. Has anyone encountered similar issues with ARM-based distributed inference clusters?

Would appreciate any insights or debugging suggestions! 🛠️

[b4rtaz]

Hello @alphadance,

I see in the CPU section that dotprod is not detected/available (🧠 CPU: neon fp16). Should this processor support it?

You can try to reduce the context size for tests: --max-seq-len 1024. Maybe this is a problem with swap.

About the network, I plan to extend the benchmark in next releases. Currently not available.

[b4rtaz]

@alphadance you can try the 0.12.8 version, this versions shows more details about the synchronization in inference mode.