Log anomaly detection advice

[TitasDas]

Posting the relevant stackoverflow question here - https://datascience.stackexchange.com/questions/123300/how-can-i-apply-nlp-nlu-methods-for-anomaly-detection-in-structured-log-data.

Would appreciate any ideas pertaining to this. Please go ahead and post there directly or here, whatever is convenient.

[zhujiem]

For production use, I would suggest feature engineering + clustering/binaryclassification. For research work, maybe pretraining or graph neural network could be promising.

[Tornadosky]

Could you elaborate on which feature engineering techniques you recommend, especially for unstructured textual logs coming from a car production plant?

Most literature I’ve seen focuses on parsing via tools like Drain, some use handcrafted templates or regular expressions, but I haven't come across many concrete examples of transforming unstructured logs into usable features for classification or clustering.

Some specific questions:

• What kinds of features (e.g., TF-IDF, word embeddings, event frequency vectors, log key sequences) tend to work well in practice?

• Are there domain-agnostic techniques you’ve seen succeed across various types of logs?

[TitasDas]

Thank you for your answer @zhujiem . Could you explain the reasoning or decision making process as to how you would choose one method over another? Specially pertaining to the type of data shared in the question.

[TitasDas]

Could you elaborate on which feature engineering techniques you recommend, especially for unstructured textual logs coming from a car production plant?

Most literature I’ve seen focuses on parsing via tools like Drain, some use handcrafted templates or regular expressions, but I haven't come across many concrete examples of transforming unstructured logs into usable features for classification or clustering.

Some specific questions:

• What kinds of features (e.g., TF-IDF, word embeddings, event frequency vectors, log key sequences) tend to work well in practice?

• Are there domain-agnostic techniques you’ve seen succeed across various types of logs?

@Tornadosky hard to generalize like this. Is there a sample of this type of data that you have available and can share? I can then try to answer your questions.

[mmantyla]

Some specific questions:

• What kinds of features (e.g., TF-IDF, word embeddings, event frequency vectors, log key sequences) tend to work well in practice?
• Are there domain-agnostic techniques you’ve seen succeed across various types of logs?

There are some papers that benchmark different log processing options. There seems to be no universal solution on what works best. Meaning: I would start with simplest thing that might work and then improve from there if performance is no good.

The papers

With supervised ML
https://link.springer.com/article/10.1007/s10664-023-10364-1

Unsupervised ML
https://ieeexplore.ieee.org/abstract/document/10684630

With DL
https://link.springer.com/article/10.1007/s10664-024-10533-w