Slow Loading of Larger Datasets in Dash AG Grid

[BalaNagendraReddy]

Description: When deploying our Dash application on an Apache server, loading Parquet files with 3 million rows in Dash AG Grid takes 10-20 minutes. This severely impacts user experience and usability.

Observed Issue:
The initial page load is taking excessive time.
Caching may not be a viable solution since calculations need to be performed dynamically based on user selection.

Expected Behavior:
The page should load significantly faster even with large datasets.
Parquet file processing should be optimized for smoother interaction.

Steps to Reproduce:
Deploy the Dash application on an Apache server.
Load a Parquet file containing 3 million rows into Dash AG Grid.
Observe page loading time (~10-20 minutes).

from dash import Dash, html
import dash_ag_grid as dag
import polars as pl

df = pl.read_parquet("green_tripdata_2019-01.parquet")
# print(df.describe())
rowData = df.to_dicts()
defaultColDef = {
    "floatingFilter": True,
    "wrapHeaderText": True,
    "autoHeaderHeight": True,
    "flex": 1,
}


app = Dash(__name__)

app.layout = html.Div(
    [
        html.H6("Dash AG Grid"),
        dag.AgGrid(
            rowData=rowData,
            columnDefs=[
                {"field": "VendorID"},
                {"field": "lpep_pickup_datetime"},
                {"field": "lpep_dropoff_datetime"},
                {"field": "total_amount"},
                {"field": "payment_type"},
                {"field": "trip_type"},
            ],
            columnSize="sizeToFit",
            defaultColDef=defaultColDef,
            className="ag-theme-alpine",
            enableEnterpriseModules=True,
        ),
    ]
)

if __name__ == "__main__":
    app.run(debug=True)

Link to download parquet file:
https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2019-01.parquet

Floating filter is also not working as expected.

In VectorID Column it is not showing vector id values to filter.

Any suggestions will be appreciated.

[BSd3v]

Hello @BalaNagendraReddy,

With that many rows, it is recommended to use things like infinite rowModels, or serverside models with enterprise.

Other things you can do is to introduce pagination to help present the data.