Finding Top K Neighbours

[aycabayraktar]

Hi everyone,

I need to find the top K neighbour of the best motif pair. Actually, I can use the right or left index but how I am going to decide how far away the right index subsequence is from the original motif

[aycabayraktar]

In addition, as I understood stumpy.motifs code sorts the top-k neighbors of the first best motif pairs' minima. But as you can see, it ignores some values, why?

[seanlaw]

Hi @aycabayraktar Thank you for your question and welcome to the stumpy community!

I can use the right or left index but how I am going to decide how far away the right index subsequence is from the original motif

For the right/left index, you'll have to re-compute the distances based on the subsequence index. So, let's say your motif was located at idx = 10 and the subsequence index for the right nearest neighbor is located at idx = 127, then you can compute the z-normalized distance (this is the default STUMPY distance) via:

motif_subseq = T[10 : 10 + m]
motif_subseq = stumpy.core.z_norm(motif_subseq)
nn_subseq = T[127 : 127 + m]
nn_subseq = stumpy.core.z_norm(nn_subseq)
distance = np.linalg.norm(motif_subseq - nn_subseq)

But as you can see, it ignores some values, why?

What do you mean by "ignores some values"? What values do you feel have been ignored?

I recommend reading through the stumpy.motifs docstring to get a better idea of what is calculated and what results are stripped/filtered away

[aycabayraktar]

Hi @seanlaw , thank you for your answer. Let me explain:

1. Time series that I used is this: nasdaq_data = yf.Ticker("^IXIC").history(period="5y", interval="1d")
2. mp is here: mp = stumpy.stump(nasdaq_data['Close'], m=20)
3. I used stump.motifs as you can see on the previous answer.
4. I took the result index array and searched in the mp. As I can see, 2nd and 6th elements (508 and 819 indexes respectively) of array are same with stumpy.motifs value results, this is what I expected, because they are also neighbors of index 151.
5. So I looked for the other indexes who are neighbor of 151. condition(mp[:,1]8 == 151). The result set is (48,508,819). 508 and 819 are ok they are in the result set of stumpy.motifs but 48 is not in the result set. why? even the distance value smaller than some elements in the dataset.

[seanlaw]

@aycabayraktar It's very difficult to guess at what the issue might be. Can you provide a minimum-reproducible code example? What you've shown above is not sufficient. This way, others can chime in and help. Otherwise, without code showing the process from start to finish, it is hard to get your point and to reproduce your problem as it may be an issue with understanding the documentation.

[aycabayraktar]

@seanlaw thank you. I am sharing the code snippet.

import yfinance as yf
# Define tickers for NASDAQ and get data
nasdaq_data = yf.Ticker("^IXIC").history(period="5y", interval="1d")
# matrix profile for 20 days window length
mp = stumpy.stump(nasdaq_data['Close'], m=20)
#top K motifs of the best
stumpy.motifs(nasdaq_data['Close'], mp[:, 0],max_distance=2.0)
#index = 151 values which are the motifs best pairing with motif_idx
condition = mp[:, 1] == 151
indexes = np.where(condition)[0]
#value of index 48 which is smaller than some values in the result set of stumpy.motifs
mp[48]

[seanlaw]

@aycabayraktar I think you may have misunderstood the meaning of the matrix profile data structure (and its relationship with motifs).

#index = 151 values which are the motifs best pairing with motif_idx
condition = mp[:, 1] == 151
indexes = np.where(condition)[0]
#value of index 48 which is smaller than some values in the result set of stumpy.motifs
mp[48]

No, condition = mp[:, 1] == 151 are NOT the motifs best pairing with motif_idx. All that the (first two columns of the) matrix profile tell you is the ONE-NEAREST-NEIGHBOR (in both distance and subsequence index location) to that reference subsequence. Thus, multiple subsequences can have, say, subsequence T[151: 151 + m] as its one-nearest neighbor BUT it is also possible that NONE of them are actually the one-nearest neighbor to T[151:151 + m].

If you really want to know the full order of subsequences (by index location) ranked from "closest" to "farthest" away then you'll need to compute the distance profile using stumpy.mass.

[aycabayraktar]

@seanlaw I think, I couldn't be able to define clearly.
Matrix profile keeps the minimum result of distance profile for the related subsequences. That's ok. So, for index 151, the minimum value of distance profile is with 507. So the value is stored in 151th index (1.column) and the neighbor index stored in (2.column).
What I mentioned above is not a rule or understanding actually, it was just a logical questioning. Actually my point was why some values are not be considered in motif list. I think, it also compares the zones after ranking and delists the indexes which are very closed to the already found neighbor.

distance_profile = stumpy.mass(nasdaq_data['Close'][151:171], nasdaq_data['Close'])

np.argsort(distance_profile)[:50]

Thanks for your explanations.

[seanlaw]

I think, it also compares the zones after ranking and delists the indexes which are very closed to the already found neighbor.

Yes, similar to the distances computed in the matrix profile, the distances that are computed by stumpy.motifs also excludes all subsequences that are within each reference subsequence's exclusion zone.

However, when you compute the distances using stumpy.mass, ALL of the distances are returned and it is up to you to ignore the exclusion zone! Failure to exclude the subsequences within the exclusion zone will likely lead to unwanted results so please take care! This is why stumpy.motifs was created as it takes care of all of this for you. Again, please note that all matrix profile calculations ALWAYS/MUST ignore all exclusion zones by default (according to the original matrix profile papers).