DynamoDB usage higher on v3 than on v2

[charles-slc]

We have a simple (single shard/single worker) .NET Kinesis consumer (using the https://github.com/awslabs/amazon-kinesis-client-net) that we've just upgraded to v4 of the amazon-kinesis-client-net; which includes v3 of the KCL library.

I had noticed the following in the v3 Readme

Optimized DynamoDB RCU usage

• KCL 3.x optimizes DynamoDB read capacity unit (RCU) usage on the lease table by implementing a global secondary index with leaseOwner as the partition key. This index mirrors the leaseKey attribute from the base lease table, allowing workers to efficiently discover their assigned leases by querying the index instead of scanning the entire table.
• This approach significantly reduces read operations compared to earlier KCL versions, where workers performed full table scans, resulting in higher RCU consumption.

However, I see significantly higher SCAN usage in addition to the new QUERY usage in the DynamoDB lease table for our app.
You can see the before/after usage below. New version went live shortly after 8am local time.

Is there some setting I can change to reduce DDB usage?

[akidambisrinivasan]

Hello @charles-slc - Thank you for reporting this issue. Currently our new LeaseAssignmentManager performs a parallel scan of the lease table. Which seems to have caused a regression for small number of leases. While there is currently no way to tweak the DDB usage in KCLv3. We are currently looking at improving the code to potentially pick the number of segments based on the table size which should avoid this regression.

[charles-slc]

Thank you, note that I also have an open case with AWS Support, 173204859800082.