Finding patterns based on a condition

[sword134]

Hello,
I'd like to know if there is a way in Stumpy to find patterns before a condition occurs. Say for example I want to analyze a data series and want to find a pattern before something happens. For example I want to find a pattern before a big decrease happens in the data or increase. Is this currently possible in Stumpy if I dont know what this pattern looks like? The idea is finding the pattern that leads to this decrease or increase without knowing it beforehand.

[seanlaw]

@sword134 Thanks for the question and welcome to the STUMPY community. While there isn't any built in function that will do this for you, it should be relatively straightforward to do this in a few different ways with STUMPY (assuming that you have a desired window size in mind). If you have a time series where there are two big decreases in the data that are somehow related (let's call these A and B, then you can set those regions to np.nan and then compute the matrix profile for this time series. Then, you can look at the matrix profile indices for the region that happens before A and see if any of their one-nearest-neighbors is located in the region that happens right before B.

If this is insufficient, you can also perform an AB-join between the region before A and the region before B and decide whether there are any patterns based on whether or not any of the distances are below some user defined distance threshold.

I hope that helps!

[sword134]

@sword134 Thanks for the question and welcome to the STUMPY community. While there isn't any built in function that will do this for you, it should be relatively straightforward to do this in a few different ways with STUMPY (assuming that you have a desired window size in mind). If you have a time series where there are two big decreases in the data that are somehow related (let's call these A and B, then you can set those regions to np.nan and then compute the matrix profile for this time series. Then, you can look at the matrix profile indices for the region that happens before A and see if any of their one-nearest-neighbors is located in the region that happens right before B.

If this is insufficient, you can also perform an AB-join between the region before A and the region before B and decide whether there are any patterns based on whether or not any of the distances are below some user defined distance threshold.

I hope that helps!

Thank you for the quick reply @seanlaw !
Is there a way to do it with a multivariate dataset? Also would it be possible to provide some sample code and perhaps add it to the documentation for future use. I feel like this is a very important use case of stumpy that hasnt had any light shed on it.

[seanlaw]

Is there a way to do it with a multivariate dataset?

Hmm, I'm not sure about this since things become more complex when we add more dimensions. We do have some multi-dimensional support in mstump but that area of research is still evolving. Perhaps, you can play with this and report back?

Also would it be possible to provide some sample code and perhaps add it to the documentation for future use. I feel like this is a very important use case of stumpy that hasnt had any light shed on it.

I'd love to but the hardest part is tracking down a good data set that will easily convey the core concept(s) there. In general, we've tried using contrived data examples in our tutorials before (i.e., where you insert known patterns into some random noise) and it has typically caused more confusion than good. Unfortunately, this ends up being more work (i.e., questions) for the developers and maintainers. Ultimately, while it may be an interesting use case, what you are asking isn't something that we've actively explored and is therefore hard to endorse any particular approach. Our tutorials always try to reproduce one of the figures from the Matrix Profile paper but I've not come across any good data that contains clear/consistent leading indicators.

Perhaps, you can reach out to the matrix profile authors and ask if they have any good examples that we might be able to use for finding leading indicators?