The correct position of 2 layernorm weights.

[Blacknana]

In your code, the mappings of 2 layernorm weights are:

'layers.mlp.linear_fc1.layer_norm_bias': 'model.layers.{lnum}.mlp.input_layernorm.bias',
'layers.mlp.linear_fc1.layer_norm_weight': 'model.layers.{lnum}.mlp.input_layernorm.weight',
'layers.self_attention.linear_qkv.layer_norm_bias': 'model.layers.{lnum}.post_attention_layernorm.bias',
'layers.self_attention.linear_qkv.layer_norm_weight': 'model.layers.{lnum}.post_attention_layernorm.weight',

For LlamaForCausalLM, model.layers.{lnum}.post_attention_layernorm.weight marks the position before MLP block，while model.layers.{lnum}.mlp.input_layernorm.weight marks the position before Attention block. I wonder how you determine if their location is correct.