o Use numpy unique instead of polars

[rajeeja]

No description provided.

[philipc2]

@rajeeja

Polars is significantly faster than NumPy. I'm curious to why we would want to switch back?

[rajeeja]

@rajeeja

Polars is significantly faster than NumPy. I'm curious to why we would want to switch back?

Agreed, polars operates in a multi-threaded manner, this avoids convertion overhead, NumPy arrays → Polars → NumPy multiple times, also there is no convertion between DataFrame representations. Depends on the optimizing for the problem size we want to tackle.

[rajeeja]

this looks cleaner also, but we can go with the polars version ignoring some of the overhead for small/medium problems based on our target n_pix size.

[rajeeja]

Running a benchmark (Pixel size ) with the two versions would be good.

[philipc2]

From an email with @erogluorhan and @florianziemen last Month.

I was able to significantly optimize the Numpy unique calls with Pandas; here are some figures:

For a resolution level of 10 (i.e. n_side==1024, n_pixels== ~12.6M):

Numpy unique took 1 min 34 secs in my machine, while the pandas equivalent (calculations of uniques nodes and inverse indices comboned) took only ~16.4 seconds.

With Polars, its only a couple of seconds compared to Pandas.

[rajeeja]

From an email with @erogluorhan and @florianziemen last Month.

I was able to significantly optimize the Numpy unique calls with Pandas; here are some figures:
For a resolution level of 10 (i.e. n_side==1024, n_pixels== ~12.6M):
Numpy unique took 1 min 34 secs in my machine, while the pandas equivalent (calculations of uniques nodes and inverse indices comboned) took only ~16.4 seconds.

With Polars, its only a couple of seconds compared to Pandas.

wow, just tried it and found as you mention, for 10M pixel corner coords.

NumPy unique() time: 47.6 secs

Polars unique() + join time: 5.26 secs