Description
When fetching Historical data via the Market Data API, there is no way of manually paginating the data (via the page tokens as specified in the docs), and there is no way of seeing the progress of the internal paginations.
This is pretty problematic:
- It can lead to explosions in the amount of memory used (for example, if requesting several months of trades/quotes, since everything is appended into a gigantic slice)
- It can lead to clients not being able to indicate progress on the request (fetching more than a few weeks of data can take a large amount of time and the program pretty much just hangs...)
In the v2 version of the module, there was a GetxxxAsync version of each of the functions, which let the user stream the entities over a channel. This solved both of the above issues:
- Data wasn't appended to a giant slice, so the memory could be managed more by the user (you can choose to append to a slice in memory, or you can write to a DB/File/etc. and not hold on to the memory)
- Progress could be reported as the entities were streamed to the consumer of the channel.
I understand that in v3 the Async functions were removed (which sucks for me, I was using them heavily to stream historical data through my systems for simulations and backtesting), but since the PageToken (on the request) and NextPageToken (on the response) values aren't exposed on the client, I cannot even implement pagination/streaming manually using the official module, which is a huge problem...
Can we either:
- Return the Async versions of the functions to the module, OR
- Expose the PageToken on the request structs and NextPageToken on the function responses, so that pagination can be handled manually