Documentation about create/aplly syslog filter/rules

[ccesario]

Hello,

I would like how to create filters/rules to generic Syslog events. I did not found documentation about it.

Currently I have this kind of Syslog

logx.syslog.message 

<14>1 2025-04-09T08:55:51.766313-03:00 SRV-xxxxx Veeam_MP - - [origin enterpriseId="23423423"] [categoryId=0 instanceId=190 JobSessionID="e3df7013-3968-448a-b605-xxxxxx" JobID="d4cb0be1-ff01-446c-a3af-xxxxxx" JobResult="0" JobType="0" Platform="0" WillBeRetried="False" VbrHostName="SRV-xxxxx" VbrVersion="1.1.1.1" Version="1" Description="Backup job 'JOBNAME_XYZ' finished with Success. All objects have been backed up successfully."]

And I would like to split it into fields. How to I can do it in UTMstack ?

Regards

Carlos

[agauttam]

Hi Carlos,

You have two ways to parse the logs.

1. Using logtash integration
2. Using Json integration

After parsing you can create a rule base on defined parameter or else for syslog you can define rule like logx.syslog.message "include" than keyword which you want to check in syslog. But the better way is to parse the syslogs properly using logtash and than create the rules

Please refer the below link for logtash parser.
https://github.com/yetanothermightytool/logstash-elastic-veeam-syslog/tree/main/pipeline

Regards,
Arun

[ccesario]

Hi @agauttam, Yes, I know that using grok patterns, but how can I apply it inside UTMStack ?

Let me deails what I already tried
Currently Im testing patterns in this tool!

https://grokconstructor.appspot.com/do/match#result

The log data
<14>1 2025-04-09T08:55:51.766313-03:00 SRV-xxxxx Veeam_MP - - [origin enterpriseId="23423423"] [categoryId=0 instanceId=190 JobSessionID="e3df7013-3968-448a-b605-xxxxxx" JobID="d4cb0be1-ff01-446c-a3af-xxxxxx" JobResult="0" JobType="0" Platform="0" WillBeRetried="False" VbrHostName="SRV-xxxxx" VbrVersion="1.1.1.1" Version="1" Description="Backup job 'JOBNAME_XYZ' finished with Success. All objects have been backed up successfully."]

The pattern
<%{POSINT:priority}>%{NUMBER:version} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:sourcehost} %{WORD:product} - - \[%{DATA:origin}\] \[%{GREEDYDATA:data}\]

And this work

Using UTMStack

I create a new Pattern Pipeline in Sysylog Generic

With the same content that I previous tested

But his appear does not work

Regards
Carlos

[agauttam]

Hi Carlos,

First install the Logtash, write the complete logtash parser where you can test bulk logs. Your parser is not parsing entire syslogs thats why you are getting grokparsefailure. you can try to copy the entire parser from reference link which i shared in my last response

Regards,
Arun

[agauttam]

Try with the below file may be it will fulfill your requirement
https://github.com/yetanothermightytool/logstash-elastic-veeam-syslog/blob/main/pipeline/logstash-veeam-syslog.conf

Make sure remove input and output section.

[ccesario]

@agauttam
Yes I already tried the entire parser https://github.com/yetanothermightytool/logstash-elastic-veeam-syslog/blob/main/pipeline/logstash-veeam-syslog.conf I alreadytest it before ask in the forum ...
And I receive the grokparsefailure :)

Regards

[agauttam]

The correct way to test this is to copy the existing, pre-configured syslog parser in your system and modify the syslog parser pipeline. Do not add a new one.

If it doesn’t work as expected, I recommend setting up and testing the configuration in Logstash separately before applying it to UTMStack.

Additionally, you can log in to the UTMStack command line and check the following path before and after making any modifications:

/utmstack/logstash/pipelines/syslog/
It's best practice to back up the filter.conf file by copying it to another location before making any changes.

Regards,
Arun

[ccesario]

Sure, even this, this does not work as expected

I have created new filter file.

And remove the config in UTMStack

Im getting error

[agauttam]

you dont need to create a new filter file you need to delete clean the already existing filter and update the file like below. but before that take the backup of existing file.

[ccesario]

Hi @agauttam,

Yes, replacing the default syslog filter by other, it works

But, just to know to correct mode to create new "filters" inside UTMStack!?

Regards

Carlos

[ccesario]

Hi @agauttam,

Have you any tip about crete a new "filter" without replace the default Syslog ?

Regards
Carlos

[agauttam]

Hi Carlos,

You need to install logtash and tune on it before replacing the filter in UTMStack.

You can easily use your sample logs of Veeam in logtash, like direct forward as syslog or using file.

As per my understanding UTMStack is build on top of Elastic (OpenSearch) and Logtash.

Regards,
Arun

[ccesario]

Hi @agauttam,
Thank you by you feedback.
Yes, I understood, that I need write the filters/rules based in Logstash.
But I think that replace the Default UTMStack will not a correct way, and yes, create new filter and usage it.

Regards

Carlos

[agauttam]

Hi Carlos,

You can replace the default filter with your own, or use multiple conditions in your custom filter. If none of your defined conditions match, the logs will be parsed using the default filter at the end.

In case your custom filter conditions don't successfully parse the logs, the default filter will act as a fallback.

Regards,
Arun

[ccesario]

Hi @agauttam, yes. I understand, but I think that is not good way.

Maybe this #840 answer my question and add the possibility to add new parser.

Regards
Carlos

[agauttam]

Hi Carlos,

You can refer to #750 to understand the complexity involved in parsing logs. If it were truly that simple, companies like Microsoft and Splunk wouldn’t face ongoing challenges when integrating diverse log sources into their SIEM solutions.

In the case of UTMStack, log parsing is handled by Logstash, as per the platform's architecture. This component is responsible for processing and structuring incoming logs, but even with such tools in place, managing the variety and inconsistency of log formats remains a significant technical hurdle.

Regards,
Arun