Positional Encoding and its Averaging

[robhta]

I’ve been reviewing the current implementation and had a question regarding using positional encoding. Specifically:

Is it possible that positional encoding has no effect in the current implementation?

From what I’ve observed, averaging seems to nullify the impact of the positional encoding. Since positional encodings inject information about token order, averaging across positions might remove this order-specific signal.

I have written down the following:


I’d appreciate your thoughts or clarification on this, especially if I’m missing something in how the encoding is being applied or preserved.

Thanks in advance!