🚀 Feature Request

Add Streaming Support for Hugging Face Parquet Datasets Similar to Chunk-Based Streaming

Motivation

Currently, the entire dataset must be downloaded before indexing, resulting in significant waiting time before streaming can begin. This approach is inefficient, especially for datasets with a large number of chunks.

Pitch

The proposed solution is to read only the metadata of all Parquet files using byte-range requests, allowing for quick indexing. Streaming can then proceed similarly to LitData's chunk-based streaming approach.

Reference: Introduction to Parquet Format