Pattern Search

[t4fast]

Hi,
I have implemented Fast Pattern Searching Tutorial on some data.
The Predefined pattern is in orange color. I am not happy with matched/identified pattern;

I was looking for highs and lows to be at the same level (marked in red), however, the matched pattern didn't respect the structure.

Is there a way to either add weight on some points ?! or any other way to solve this ?

Thanks in advance.
Regards

[seanlaw]

@t4fast Welcome to the STUMPY community! Can you tell me what version of STUMPY you are currently using and what command(s) you are using to create the plots above?

[t4fast]

HI Sean, I have used

• distance_profile = stumpy.core.mass(Q_df["Price"], T_df["Price"])
• idx = np.argmin(distance_profile)
• Q_z_norm = stumpy.core.z_norm(Q_df.values)
• T_z_norm = stumpy.core.z_norm(T_df.values[idx:idx+len(Q_df)])

I am using version ..

C:\Users\User>pip install stumpy
Requirement already satisfied: stumpy in c:\python\python38\lib\site-packages (1
.8.0)
Requirement already satisfied: scipy>=1.5 in c:\python\python38\lib\site-package
s (from stumpy) (1.6.3)
Requirement already satisfied: numba>=0.48 in c:\python\python38\lib\site-packag
es (from stumpy) (0.53.1)
Requirement already satisfied: numpy>=1.15 in c:\python\python38\lib\site-packag
es (from stumpy) (1.19.0)
Requirement already satisfied: setuptools in c:\python\python38\lib\site-package
s (from numba>=0.48->stumpy) (47.1.0)
Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in c:\python\python38\l
ib\site-packages (from numba>=0.48->stumpy) (0.36.0)

[seanlaw]

Thanks for the info. So, I'm not exactly sure that this will address your issue but stumpy.core.mass(Q_df["Price"], T_df["Price"]) will z-normalize your subsequences first before computing the Euclidean distance(s). In your plots above, are you plotting the z-normalized subsequences or the raw time series?

It sounds like what you may need to do is turn off z-normalization and you can do this with stumpy.core.mass_absolute():

distance_profile = stumpy.core.mass_absolute(Q_df["Price"], T_df["Price"])
idx = np.argmin(distance_profile)
Q = Q_df.values  
T = T_df.values[idx:idx+len(Q_df)]

Please note that starting in STUMPY v.1.9.0, stumpy.core.mass can be called directly as stumpy.mass and you can turn off z-normalization by doing stumpy.mass(Q, T, normalize=False). In other words, we've simplified the API significantly and while all three of the following commands will become equivalent but the first line is preferred:

stumpy.mass(Q, T, normalize=False)  # Preferred starting in STUMPY v1.9.0
stumpy.core.mass(Q, T, normalize=False)  # Deprecated starting in STUMPY v.1.9.0
stumpy.core.mass_absolute(Q, T)  # Deprecated starting in STUMPY v.1.9.0

[t4fast]

Thanks Sean for the reply
I think I must use the z-normalization as the pattern is generic; it has different value ranges than the data set.
I have tried the mass_absolute, and match identification was worse, I was expecting it not to give any match, but it did.

It is clear that it is matching (ref to first pic above). However, the structure is not maintained. I hope I was able to explain this point.

[seanlaw]

From my (limited) perspective, it does look like the shape is maintained, which is what I would expect with z-normalization. However, this also suggests that no other subsequence (when z-normalized) is a better match to the orange curve than the blue one. In addition to the idx = np.argmin(distance_profile) have you tried looking at the top-K closest matches:

k = 5
top_k_indices = np.argsort(distance_profile)[:5]

[NimaSarajpoor]

@t4fast

However, the structure is not maintained. I hope I was able to explain this point.

As @seanlaw mentioned, there is probably no better matching. What you can do is to explore the distance profile you calculated. You should probably see that the distance between the two patterns you showed earlier (between the query and the other subsequence) is smallest.

I think I must use the z-normalization as the pattern is generic

Just wanted to point out something here: If you have a trend in your time series, it means the average is changing, and if the magnitude is important (it seems it is as the data is price!), just be careful in using z-normalization.

Furthermore, you mentioned that the match identification was worse without z-normalization. I think that does NOT necessarily mean that you should do z-normalizing.