Supporting passing in weights

[rofinn]

It might be nice to support a lower level fit and transform API on top of just cov. This would allow folks to pass in an existing covariance matrix that they'd like to apply an estimation method to (e.g., a weighted covariance).

[mateuszbaran]

I don't quite understand how would that work. Implemented formulas for covariance shrinkage assume normal covariance and a specific shrinkage target. Wouldn't using weighted covariance lead to wrong shrinkage coefficients?

[rofinn]

Hmmm, good point. I guess the R corpcor just lets you pass in the weights with a data matrix rather than precomputed values.

[mateuszbaran]

StatsBase.jl has its own approach to weights and it looks nicer. I think it would be good to have a similar design here.

[nickrobinson251]

Supporting weights, following the StatsBase API of cov(ce::CovarianceEstimator, X::AbstractMatrix, w::AbstractWeights), would be really useful :)

[mateuszbaran]

Yes, it would be very useful, but I still don't know if it's even possible to have generic implementation for all types of weights. Frequency weights should be relatively easy to support though. What kind of weights do you need?

[nickrobinson251]

I believe AnalyticWeights. But also I am not sure weighted Lediot-Wolf is possible (I was hoping it was)

[mateuszbaran]

Unfortunately, I don't know how to modify Ledoit-Wolf to make it work with AnalyticWeights. If you find a paper about it, I will definitely like to see it.

[nickrobinson251]

My colleague tells me it is not possible (or at least the proof in the LW paper breaks down when weights are added) ...but i've not done the working myself nor seen a paper on it.

[rofinn]

NOTE: a potential work around could be to collect a weighted sampling first and then estimate the covariance from that.