single time series Distance Matrix with STUMPY

[Jalanjii]

Hi Sean, I have a single time series. I want to build the Distance Matrix as you did in this tutorial, https://stumpy.readthedocs.io/en/latest/Tutorial_The_Matrix_Profile.html, there is a manual brute force way to do that as it is explained, but the Stumpy way I don't find it as far as my reading has gone. For mpdist, it is useless I think in this case, right? I want to calculate the distance matrix from a single time series, I am not interested the profile for now.

I am aware of the paper about subsequence clustering and the randomness involved. Regardless, I just want to try to do single time series clustering, with the Distance Matrix calculated as in the above tutorial to be fed into HDBSCAN.

[NimaSarajpoor]

@Jalanjii
If you do not mind, I am going to share with you my thoughts. Btw, I am not sure which paper you were referring to when you said:

I am aware of the paper about subsequence clustering and the randomness involved

Are you talking about this paper?

---

Okay...let's say you do not care if clustering of subsequences of a single time series is meaningless or not. You just want to perform a clustering on the subsequences (with length m) and see the results. Right?

So, Your main time series T contains len(T) - m + 1 subsequences (with length m) and you are going to treat them as your observations (your final goal is to cluster them).

k = len(T) - m + 1 # number of subsequences of length `m` in `T` 
# So, you want to cluster these `k` subsequences

Now, you need to find the distance matrix D, which is a k-by-k matrix. D[i, j] is the distance between S_i = T[i : i +m] and S_j = T[j : j + m].

But, how do you get the distance between S_i and S_j? We need to use distance measure function d(.), where d(S_i, S_j) is the distance between S_i and S_j. If I understand correctly, mpdist is simply a distance function. So:

# distance between `S_i` and `S_j` using mpdist
d(S_i, S_j) = mpdist(S_i, S_j, window, percentage, ...)  # D[i, j] = d(S_i, S_j)

I think window is different than m here. The latter is the length of your observations. The former, i.e. window, is the window-size that should be considered for calculating the mpdist between two subsequences that have length m. what is a proper value for window? what should be percentage? I do not know :) You can read the mpdist paper to get some idea and then you can play with them.

from stumpy.mpdist import _mpdist_vec
k = len(T) - m + 1
D # empty 2D array with shape (k, k) 
for i in range(k):
    D[i] = _mpdist_vec(T[i: i+m], T, window, ...') 

A few notes:

• I am not 100% sure about _mpdist_vec(Q, T, ...). The description of its output is missing from docstring (@seanlaw ). But, based on the description of the function, I THINK it calculates the distance between Q and each subsequence (with length len(Q)) in T. The distance between Q and each of those subsequences is calculated by mpdist distance measure.
• The implementation I provided above may not be efficient. This is just to give you some idea.
• If there is overlap between S_i and S_j (that overlap depends on i, j and exclusion_zone), then that means there is a subsequence in S_i that appeares in S_j. If window is less than the length of that overlap, then that means one of the pairwise distances between subsequences of S_i and S_j becomes zero. And, that can affect the result. So, I THINK you should be really careful in setting exclusion_zone (so how much overlap you allow to exist between S_i and S_j) and the window and other parameters.

[NimaSarajpoor]

Btw, I think you can pass this distance matrix D as input to any clustering algorithm that can work with distance matrix. For instance, you may use K-Medoids!

[Jalanjii]

Thanks!

[seanlaw]

@Jalanjii Long-story-short, STUMPY does not provide a function/API to compute a pairwise distance matrix.

[Jalanjii]

Thanks! Can you close this issue and this one #608?