Can STUMPY/matrix profile be used for non-time series data?

[SultanOrazbayev]

I haven't read the theory/math behind the matrix profile, so I apologise if this is a silly question, but can STUMPY/matrix profile be used for non-time series data? My guess is that the underlying data must be sequential and regular, but what happens if the data is sequential, but not recorded at a regular interval?

[seanlaw]

@SultanOrazbayev Welcome to the STUMPY community and thank you for your question. So, technically, matrix profiles operate on sequential data and does not explicitly use "time" in its computation of the matrix profile. While it is somewhat assumed that your data is recorded on a regular interval, STUMPY does not complain if your data isn't actually from a regular interval (as it has no knowledge of this information). Thus, if you provide data that is not sampled on a regular interval then it is the user's responsibility to keep that in mind when interpreting the results. You can either try analyzing your data as-is OR aggregate up to common interval. In fact, I recommend just trying both and looking at the results. Hopefully, if your data isn't too big then this should be a reasonably quick analysis.

I'd be happy to discuss this more or let me know if this makes sense?

[SultanOrazbayev]

Dear Sean, thank you very much for the explanation and leading the development of STUMPY.

The concern I have is that, for example if we are dealing with time series data at daily frequency, it's easy to create rows for missing days (e.g. holidays or weekends), but this becomes very expensive for data at higher resolutions (e.g. seconds). Aggregation is one way out of that, but might obscure high-resolution patterns.

This becomes relevant for non-time series data, e.g. GIS data, where I might have high accuracy lat/lon information on some variable, e.g. taxi pickups, but with missing observations for some lat/lons (e.g. it's not possible to pick up inside a building block or a park, or some sub-parts of spaces were not yet explored). However, I do not yet have in mind any specific problem to solve with STUMPY/matrix profile in non-time series context.

[seanlaw]

The concern I have is that, for example if we are dealing with time series data at daily frequency, it's easy to create rows for missing days (e.g. holidays or weekends), but this becomes very expensive for data at higher resolutions (e.g. seconds). Aggregation is one way out of that, but might obscure high-resolution patterns.

Indeed, these are legitimate concerns but the idea of imputation of missing data is a whole other field of research on its own and is a problem that wreaks havoc on all time series analysis methods (not only matrix profile computation). As the saying goes, "garbage in, garbage out". At the end of the day, the analysis is only as good as the quality of the data and anybody who tries to make the process more efficient for you must also be making (false?) assumptions about the data. In STUMPY, we try to be as faithful to the originally published work as possible, followed by transparency/explicitness, and make as few assumptions as possible. I am certainly no expert on missing data and so we've purposely limited the scope of STUMPY to not allow the user to "shoot themselves in the foot". Hopefully, that helps.

This becomes relevant for non-time series data, e.g. GIS data, where I might have high accuracy lat/lon information on some variable, e.g. taxi pickups, but with missing observations for some lat/lons (e.g. it's not possible to pick up inside a building block or a park, or some sub-parts of spaces were not yet explored). However, I do not yet have in mind any specific problem to solve with STUMPY/matrix profile in non-time series context.

In case you missed it, this bus lat/lon discussion question may provide some inspiration or, minimally, serves as an informative discussion.

We strongly recommend going through our tutorials for more case studies.

[SultanOrazbayev]

Thank you very much, @seanlaw !