Skip to content

Commit 5dc5065

Browse files
ezmillerezmiller
authored
Lift tech.v3.datatype.functional operations (#90)
* Add ->general-types function * Add a general type :logical * Use type hierarchy in tablecloth.api.utils for `typeof` functions * Add column dev branch to pr workflow * Add tests for typeof * Fix tests for typeof * Return the concrete type from `typeof` * Simplify `concrete-types` fn * Optimize ->general-types by using static lookup * Adjust fns listing types * We decided that the default meaning of type points to the "concrete" type, and not the general type. * So `types` now returns the set of concrete types and `general-types` returns the general types. * Revert "Adjust fns listing types" This reverts commit d93e34f. * Fix `typeof` test to test for concerete types * Reorganize `typeof?` tests * Reword docstring for `typeof?` slightly * Update column api template and add missing `typeof?` * Add commment to `general-types-lookup` * Improve `->general-types` docstring * Add `general-types` fn that returns sets of general types * Adjust util `types` fn to return concrete types * Save changes to column api.clj * Save ongoing experiments with lifting * Save ongoing work on lifting * Adjust lift-ops-1 to handle any number of args with rest arg * Working `rearrange-args` fn * Save work actually writing lifted fns * Saving first attempt to writer operators * Add `percentiiles test * Adjust `rearrange-args to take new-args in option map * Unify two lift functions * Add in docstrings when present * Move lift utils into utils ns * Rename lifting namespaces * Lift some more fns * Make exclusions for ns header helper an arg * Add new operators and tests * Add ops with lhs rhs arg pattern * Lift '* * Add require to operators ns for utils * Update test to make it more complete * Lift `equals * Make test more accurate * Reorganize tests * Fix grammar * Lift 'shift * Uncomment 'or test * Lift 'normalize op * Life 'magnitude * Lifting bit manipulation ops * lift ieee-remainder * Lifting more functions * Add excludes * Lift a bunch of new functions * Alphebetize some lists * More alphebitization * Clean up * Instead of using `col` as arg conform to using `x & and `y * Temporarily disable failing test fix in 7.000-beta23 * Disable the correct test * Just some minor cleanup in op tests * Some more cleanup/reorg in op tests * Update generated operators namespace with switch from col -> x etc * Lift 'descriptive-statistics * Fix messed up test layout * Lift 'quartiles * Lift 'fill-range and a bunch of reduce operations * Lift 'mean-fast 'sum-fast 'magnitude-squared * Lift correlation fns kendalls, pearsons, and spearmans * Lift cumulative ops * cleanup
1 parent 704156a commit 5dc5065

File tree

5 files changed

+2197
-3
lines changed

5 files changed

+2197
-3
lines changed

‎src/tablecloth/column/api/lift_operators.clj

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
(ns tablecloth.column.api.lift-operators
2+
(:require [tablecloth.column.api.utils :refer [do-lift lift-op]]))
3+
4+
(def serialized-lift-fn-lookup
5+
{['*
6+
'+
7+
'-
8+
'/
9+
'<
10+
'<=
11+
'>
12+
'>=
13+
'abs
14+
'acos
15+
'and
16+
'asin
17+
'atan
18+
'atan2
19+
'bit-and
20+
'bit-and-not
21+
'bit-clear
22+
'bit-flip
23+
'bit-not
24+
'bit-or
25+
'bit-set
26+
'bit-shift-left
27+
'bit-shift-right
28+
#_bit-test ;; can't get this to work yet.
29+
'bit-xor
30+
'cbrt
31+
'ceil
32+
'cos
33+
'cosh
34+
'distance
35+
'distance-squared
36+
'dot-product
37+
'eq
38+
'equals
39+
'exp
40+
'expm1
41+
'floor
42+
'get-significand
43+
'hypot
44+
'identity
45+
'ieee-remainder
46+
'log
47+
'log10
48+
'log1p
49+
'logistic
50+
'magnitude
51+
'max
52+
'min
53+
'next-down
54+
'next-up
55+
'normalize
56+
'not-eq
57+
'or
58+
'pow
59+
'quot
60+
'rem
61+
'rint
62+
'signum
63+
'sin
64+
'sinh
65+
'sq
66+
'sqrt
67+
'tan
68+
'tanh
69+
'to-degrees
70+
'to-radians
71+
'ulp
72+
'unsigned-bit-shift-right] lift-op
73+
['kurtosis
74+
'sum
75+
'mean
76+
'skew
77+
'variance
78+
'standard-deviation
79+
'quartile-3
80+
'quartile-1
81+
'median] (fn [fn-sym fn-meta]
82+
(lift-op
83+
fn-sym fn-meta
84+
{:new-args '([col] [col options])
85+
:new-args-lookup {'data 'col
86+
'options 'options}}))
87+
['even?
88+
'finite?
89+
'infinite?
90+
'mathematical-integer?
91+
'nan?
92+
'neg?
93+
'not
94+
'odd?
95+
'pos?
96+
'round
97+
'zero?]
98+
(fn [fn-sym fn-meta]
99+
(lift-op
100+
fn-sym fn-meta
101+
{:new-args '([x] [x options])
102+
:new-args-lookup {'arg 'x
103+
'options 'options}}))
104+
['percentiles] (fn [fn-sym fn-meta]
105+
(lift-op
106+
fn-sym fn-meta
107+
{:new-args '([x percentiles] [x percentiles options])
108+
:new-args-lookup {'data 'x,
109+
'percentages 'percentiles,
110+
'options 'options}}))
111+
['shift] (fn [fn-sym fn-meta]
112+
(lift-op
113+
fn-sym fn-meta
114+
{:new-args '([x n])
115+
:new-args-lookup {'rdr 'x
116+
'n 'n}}))
117+
['descriptive-statistics] (fn [fn-sym fn-meta]
118+
(lift-op
119+
fn-sym fn-meta
120+
{:new-args '([x stats-names stats-data options]
121+
[x stats-names options]
122+
[x stats-names]
123+
[x])
124+
:new-args-lookup {'rdr 'x
125+
'src-rdr 'x
126+
'stats-names 'stats-names
127+
'stats-data 'stats-data
128+
'options 'options}}))
129+
['quartiles] (fn [fn-sym fn-meta]
130+
(lift-op
131+
fn-sym fn-meta
132+
{:new-args '([x options] [x])
133+
:new-args-lookup {'item 'x
134+
'options 'options}}))
135+
['fill-range] (fn [fn-sym fn-meta]
136+
(lift-op
137+
fn-sym fn-meta
138+
{:new-args '([x max-span])
139+
:new-args-lookup {'numeric-data 'x
140+
'max-span 'max-span}}))
141+
['reduce-min
142+
'reduce-max
143+
'reduce-*
144+
'reduce-+] (fn [fn-sym fn-meta]
145+
(lift-op
146+
fn-sym fn-meta
147+
{:new-args '([x])
148+
:new-args-lookup {'rdr 'x}}))
149+
['mean-fast
150+
'sum-fast
151+
'magnitude-squared] (fn [fn-sym fn-meta]
152+
(lift-op
153+
fn-sym fn-meta
154+
{:new-args '([x])
155+
:new-args-lookup {'data 'x}}))
156+
['kendalls-correlation
157+
'pearsons-correlation
158+
'spearmans-correlation] (fn [fn-sym fn-meta]
159+
(lift-op
160+
fn-sym fn-meta
161+
{:new-args '([x y] [x y options])
162+
:new-args-lookup {'lhs 'x
163+
'rhs 'y
164+
'options 'options}}))
165+
['cumprod
166+
'cumsum
167+
'cummax
168+
'cummin] (fn [fn-sym fn-meta]
169+
(lift-op
170+
fn-sym fn-meta
171+
{:new-args '([x] [x options])
172+
:new-args-lookup {'data 'x
173+
'options 'options}}))})
174+
175+
176+
(defn deserialize-lift-fn-lookup []
177+
(reduce (fn [m [symlist liftfn]]
178+
(loop [syms symlist
179+
result m]
180+
(if (empty? syms)
181+
result
182+
(recur (rest syms) (assoc result (first syms) liftfn)))))
183+
{}
184+
serialized-lift-fn-lookup))
185+
186+
(comment
187+
(do-lift (deserialize-lift-fn-lookup)
188+
'tablecloth.column.api.operators
189+
'tech.v3.datatype.functional
190+
'[* + - / < <= > >= abs and bit-and bit-and-not bit-clear bit-flip
191+
bit-not bit-or bit-set bit-shift-left bit-shift-right bit-test bit-xor
192+
even? identity infinite? max min neg? not odd? odd? or pos? quot rem
193+
unsigned-bit-shift-right zero?]
194+
"src/tablecloth/column/api/operators.clj")
195+
,)

0 commit comments

Comments
 (0)