Make column-based DataFrame ?

[singularperturbation]

What do you think about creating a DataFrame type that's more column-based (list of Column types) rather than row-based (seq[T] of tuples based on schema)?

Taking some of the example operations from example_01.nim:

# Existing
df.map(x => x.age).mean()

# New
df["age"].mean()

This (and other numeric operations on an individual column) should be faster because we could return a shallowCopy view of the DF's column rather than creating a new MappedDataFrame (copying the needed data) and then running the summary function on the new dataframe.

I'd like to try and do this as part of Hacktoberfest.

[narimiran]

I'd like to try and do this as part of Hacktoberfest.

Any news/progress about this maybe?

[bluenote10]

I had started working on a more traditional (i.e. column based) data frame library here. I didn't had much time recently to work on that, so it is still very far away from being usable, but I have plans to pick this up again soon(ish).

[narimiran]

Sorry to bring this up again (one whole year after the original post :)), but is there any progress with this?

It seems that kadro development is not very active. I guess the recommendation is to use NimData? Will it be further developed or your focus is on kadro?

[bluenote10]

@narimiran This is also still very high on my wishlist, but I'm just not sure when I will find the time. With kadro I'm still experimenting how I can get dynamically typed columns right (and probably use Arraymancer under the hood). Overall NimData and kadro address different use cases: NimData is iterator based with support for statically typed schemas, kadro is in-memory with dynamically typed schemas.