Sagemaker client issue

[SuchethaChintha]

when i am executing token_benchmark_ray.py we are getting below error
File "token_benchmark_ray.py", line 456, in
run_token_benchmark(
File "token_benchmark_ray.py", line 297, in run_token_benchmark
summary, individual_responses = get_token_throughput_latencies(
File "token_benchmark_ray.py", line 111, in get_token_throughput_latencies
request_metrics[common_metrics.INTER_TOKEN_LAT] /= num_output_tokens
TypeError: unsupported operand type(s) for /=: 'list' and 'int'
(SageMakerClient pid=15473) Warning Or Error: 'SageMakerRuntime' object has no attribute 'invoke_endpoint_with_response_stream'
(SageMakerClient pid=15473) None

[Tatiats7]

Hey @SuchethaChintha did you fix that ?

[vjaramillo]

This is probably due to using an older version of the Sagemaker SDK. Updating it should fix the issue.

[ryoshirahama]

It seems that this error occurs because there's an inconsistency in how INTER_TOKEN_LAT is handled between different LLM clients.

SageMaker client keeps INTER_TOKEN_LAT as a list

llmperf/src/llmperf/ray_clients/sagemaker_client.py

Line 109 in f1d6bed

metrics[common_metrics.INTER_TOKEN_LAT] = time_to_next_token

On the other hand, OpenAI client sums the latencies before returning

llmperf/src/llmperf/ray_clients/openai_chat_completions_client.py

Line 112 in f1d6bed

metrics[common_metrics.INTER_TOKEN_LAT] = sum(time_to_next_token) #This should be same as metrics[common_metrics.E2E_LAT]. Leave it here for now

I think that if you modify the source code for sagemaker_client.py as follows, it will work correctly.

metrics[common_metrics.INTER_TOKEN_LAT] = sum(time_to_next_token)

Even if you make this change, INTER_TOKEN_LAT is divided by the number of output tokens in token_benchmark_ray.py, so the correct metrics should be calculated.