MAX_BATCH_TOKENS parameter

[maioranisimone]

hello everyone, I would like to ask where I can find the maximum value to assign to max_batch_tokens parameter. I read that text-embeddings-inference cannot find this value automatically, where is this value declared in the models present on Hugging Face?
Thanks in advance.

[gerritvd]

I would also like to know more about this parameter. I just tested with all-minilm-l6-v2 on an L4 GPU and actually setting this parameter to 2048 increased throughput significantly compared to the default setting of 16K. Why does throughput decrease beyond and below this value for this model?