keep only the unique times series out of 1000 series

[lqmike]

I feel like I could either calculate whole matrix profile or using the stump.match to solve the problem, but not exactly sure. Hope you can share some thoughts, many thanks!

Problem: : I have a long list of time series say 10000 of them, some of them are highly correlated, and some are relatively unique, is there a way to utilize Stumpy to filter out the highly correlated ones and only keep relative unique ones, sort of like pick only 1 series from a cluster. so maybe with a threshold I have only 200 out of 1000 time series left. (In consensus motif example, meaning maybe I only want to keep Python & Alpaca out of the 4 series.)

Again, any idea is welcome thanks a lot!

[seanlaw]

The first thing that comes to mind is using mpdist to compute pairwise distances between all time series. There is a work-in-progress tutorial here:

https://github.com/TDAmeritrade/stumpy/blob/ff2e3095693173da264a99cb40a30a688581d177/docs/Tutorial_MPDist.ipynb

It may not be super fast but it should help

[lqmike]

The first thing that comes to mind is using mpdist to compute pairwise distances between all time series. There is a work-in-progress tutorial here:

https://github.com/TDAmeritrade/stumpy/blob/ff2e3095693173da264a99cb40a30a688581d177/docs/Tutorial_MPDist.ipynb

It may not be super fast but it should help

Thanks for the super fast reply. And as always, there is always some amazing tool in the Stumpy toolbox, let me check it out.

[seanlaw]

Thank you for your kind words but, really, most of the credit should go to the original academic researchers in Keogh's group for publishing their work.

As a high level summary, mpdist takes two time series and asks, "how similar are the top, say, 5% of the subsequences in each time series?" The percentage cutoff is up to you and this basically allows you to approximately compare time series of different lengths by comparing their subsequences (and not the full lengths). You'll still need to choose a subsequence window size though.

[JaKasb]

If the individual time series are

• non-overlapping in time and
• have the same length and
• aligned in the time axis for their start and end

then you can cluster them like any other n-dimensional object.
Only if you want to extract sliding window subsequences from the individual time series, then you benefit from the Matrix Profile algorithms.

The Paper "Matrix Profile XV: Exploiting Time Series Consensus Motifs to Find Structure in Time Series Sets" does closely match your problem. But mind the problem of outlier time series for the Ostinato algorithm.

But you can start simpler, eg by concatenating your list of time series and insert a NaN entry between each time series.
Then you can treat the unified concatenation like a normal time series.

[seanlaw]

In this case, consider using stumpy.ostinato