SQLServer Extended Event Handlers (#20229)

* poc test first pass

* log events

* logging

* run_job_loop, not start

* params correction

* rpc_events xml parsing basic

* batch_events and share utils

* timestamp and timing implementation

* event file implement

* fix file path

* return complete xml

* parse xml on client side

* time parsing and query section seperately

* convert string to bytes

* now test sqlserver parsing

* remove sqlserver parsing version

* missing statement from rpc_events

* print event payload

* fix json parsing

* add event source to event payload

* implement error events

* remove config

* test start time timestamp calculation

* make allen test check more loose

* log host and session id as well

* delete log

* delete correct log

* use resolved hostname

* try to detect ring buffer event loss

* more visibility on timestamp gaps

* do not limit max events for testing

* temp increase of max events

* fill in dbm_type based on event session name

* implement sql statement events

* implement sp statement events

* combine query completions to a single event session

* refactors

* implement attention events

* remove joined event handlers, add query start timing data

* clean up

* clean up

* more clean up

* RQT and obfuscate queries first pass

* get query completion timestamp into rqt event

* better timing data

* add more logging

* remove caching for now to get visibility for debugging

* calculate raw query signature

* normalize timestamps

* add xe_type

* fix event_name for error events

* add query_signature to non-RQT event

* refactor obfuscating logic

* clean up dead code

* consolidate more code

* normalize timestamp for timestamp filtering

* simplify timestamp filtering

* fix timestamp gap logging

* simplify event logging

* omit duration and query_start from query error RQT

* omit in XE event too

* refactors

* missed path fix

* add sql fields back

* explicitly state sql fields expected for each event session

* move raw query signature calculation

* implement configuration

* unit test first pass

* change imports

* import change

* add handlers test

* fix stub import

* don't mock event handler

* mock keys return dict

* fix tests

* timestamp mock fixes

* TimeMock class

* avoid mocking time.time

* refactors

* fix expected types in rqt event

* module end test

* space in file name!!

* add attention test

* fix attention test

* add integration test

* send events to datadog

* check if sleep makes test consistent

* debug test

* fix cursor call

* grant select to datadog user

* grant to bob

* wrong setup

* delete extra vars

* log all calls

* run check

* follow activity.py pattern

* fix event type

* debug logging

* fix config

* refactor test

* remove sleep

* enable cache, add timestamp test

* fix happy path test

* linter fixes part 1

* linters part 2

* concat strings for linter

* delete statement level event files

* Add database instance to events

* batch events for query_completion and query_errors

* fix unit test serialization and add test for checking batching logic

* add method tracking and code clean up

* add change log

* fix conditional logging

* remove timing data now that we have tracked methods

* log ANY first rqt event

* validate config

* fix import

* license fix

* validate models

* make collection interval a number, not int

* fix unit tests

* update all setup scripts to set up XE sessions

* add query visibility into error

* clean up code

* add raw query signature to query completion and error

* revert to execute with retries

* debug pipeline, only run on 2022 sqlserver

* use convert syntax for adodbapi

* add back 2019 sqlserver version

* address review comments

* delete dead code

* parse XML only once

* linter

* add configurable max events

* linter

* validate config

Author: azhou-datadog

sqlserver/assets/configuration/spec.yaml

@@ -885,7 +885,9 @@ files:
             display_default: false
     - name: collect_raw_query_statement
       description: |
-        Configure the collection of raw query statements in query activity and execution plans.
+        Configure the collection of raw query statements in query activity, execution plans, and XE events.
+        To collect raw query statements from XE events, set `xe_collection.query_completions.enabled` and 
+        `xe_collection.query_errors.enabled` to `true`.
         Raw query statements and execution plans may contain sensitive information (e.g., passwords)
         or personally identifiable information in query text.
         Enabling this option will allow the collection and ingestion of raw query statements and 
@@ -997,6 +999,75 @@ files:
       value:
         example: false
         type: boolean
+    - name: xe_collection
+      description: |
+        Configure the collection of events from XE (Extended Events) sessions. Requires `dbm: true`.
+
+        Set `collect_raw_query_statement.enabled` to `true` to collect the raw query statements for each event.
+      options:
+        - name: debug_sample_events
+          description: |
+            Set the maximum number of XE events to log in debug mode per collection. Used for troubleshooting.
+            This only affects logging when debug mode is enabled. Defaults to 3.
+          hidden: true
+          value:
+            type: integer
+            example: 3
+            display_default: 3
+        - name: query_completions
+          description: |
+            Configure the collection of completed queries from the `datadog_query_completions` XE session.
+
+            Set `query_completions.enabled` to `true` to enable the collection of query completion events.
+            
+            Use `query_completions.collection_interval` to set the interval (in seconds) for the collection of 
+            query completion events. Defaults to 10 seconds. If you intend on updating this value,
+            it is strongly recommended to use a consistent value throughout all SQL Server agent deployments.
+            
+            Use `query_completions.max_events` to set the maximum number of query completion events to process 
+            per collection. Note that SQL Server's ring buffer has a maximum of 1000 events per query, 
+            so values above 1000 will still be capped at 1000 by the database engine. Defaults to 1000.
+          value:
+            type: object
+            properties:
+            - name: enabled
+              type: boolean
+              example: false
+            - name: collection_interval
+              type: number
+              example: 10
+              display_default: 10
+            - name: max_events
+              type: integer
+              example: 1000
+              display_default: 1000
+        - name: query_errors
+          description: |
+            Configure the collection of query errors from the `datadog_query_errors` XE session.
+
+            Set `query_errors.enabled` to `true` to enable the collection of query error events.
+            
+            Use `query_errors.collection_interval` to set the interval (in seconds) for the collection of 
+            query error events. Defaults to 10 seconds. If you intend on updating this value,
+            it is strongly recommended to use a consistent value throughout all SQL Server agent deployments.
+            
+            Use `query_errors.max_events` to set the maximum number of query error events to process 
+            per collection. Note that SQL Server's ring buffer has a maximum of 1000 events per query, 
+            so values above 1000 will still be capped at 1000 by the database engine. Defaults to 1000.
+          value:
+            type: object
+            properties:
+            - name: enabled
+              type: boolean
+              example: false
+            - name: collection_interval
+              type: number
+              example: 10
+              display_default: 10
+            - name: max_events
+              type: integer
+              example: 1000
+              display_default: 1000
     - name: deadlocks_collection
       description: |
         Configure the collection of deadlock data.

sqlserver/changelog.d/20229.added

@@ -0,0 +1,2 @@
+Added SQLServer Extended Event Handlers
+

sqlserver/datadog_checks/sqlserver/config.py

@@ -57,6 +57,7 @@ def __init__(self, init_config, instance, log):
         self.activity_config: dict = instance.get('query_activity', {}) or {}
         self.schema_config: dict = instance.get('schemas_collection', {}) or {}
         self.deadlocks_config: dict = instance.get('deadlocks_collection', {}) or {}
+        self.xe_collection_config: dict = instance.get('xe_collection', {}) or {}
         self.cloud_metadata: dict = {}
         aws: dict = instance.get('aws', {}) or {}
         gcp: dict = instance.get('gcp', {}) or {}

sqlserver/datadog_checks/sqlserver/config_models/instance.py

@@ -347,6 +347,36 @@ class SchemasCollection(BaseModel):
     max_execution_time: Optional[float] = None
 
 
+class QueryCompletions(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    collection_interval: Optional[float] = Field(None, examples=[10])
+    enabled: Optional[bool] = Field(None, examples=[False])
+    max_events: Optional[int] = Field(None, examples=[1000])
+
+
+class QueryErrors(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    collection_interval: Optional[float] = Field(None, examples=[10])
+    enabled: Optional[bool] = Field(None, examples=[False])
+    max_events: Optional[int] = Field(None, examples=[1000])
+
+
+class XeCollection(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    debug_sample_events: Optional[int] = None
+    query_completions: Optional[QueryCompletions] = None
+    query_errors: Optional[QueryErrors] = None
+
+
 class InstanceConfig(BaseModel):
     model_config = ConfigDict(
         validate_default=True,
@@ -406,6 +436,7 @@ class InstanceConfig(BaseModel):
     tags: Optional[tuple[str, ...]] = None
     use_global_custom_queries: Optional[str] = None
     username: Optional[str] = None
+    xe_collection: Optional[XeCollection] = None
 
     @model_validator(mode='before')
     def _initial_validation(cls, values):

sqlserver/datadog_checks/sqlserver/data/conf.yaml.example

@@ -643,7 +643,9 @@ instances:
         #
         # keep_identifier_quotation: false
 
-    ## Configure the collection of raw query statements in query activity and execution plans.
+    ## Configure the collection of raw query statements in query activity, execution plans, and XE events.
+    ## To collect raw query statements from XE events, set `xe_collection.query_completions.enabled` and 
+    ## `xe_collection.query_errors.enabled` to `true`.
     ## Raw query statements and execution plans may contain sensitive information (e.g., passwords)
     ## or personally identifiable information in query text.
     ## Enabling this option will allow the collection and ingestion of raw query statements and 
@@ -797,6 +799,42 @@ instances:
     #
     # propagate_agent_tags: false
 
+    ## Configure the collection of events from XE (Extended Events) sessions. Requires `dbm: true`.
+    ##
+    ## Set `collect_raw_query_statement.enabled` to `true` to collect the raw query statements for each event.
+    #
+    # xe_collection:
+
+        ## @param query_completions - mapping - optional
+        ## Configure the collection of completed queries from the `datadog_query_completions` XE session.
+        ##
+        ## Set `query_completions.enabled` to `true` to enable the collection of query completion events.
+        ##
+        ## Use `query_completions.collection_interval` to set the interval (in seconds) for the collection of 
+        ## query completion events. Defaults to 10 seconds. If you intend on updating this value,
+        ## it is strongly recommended to use a consistent value throughout all SQL Server agent deployments.
+        ##
+        ## Use `query_completions.max_events` to set the maximum number of query completion events to process 
+        ## per collection. Note that SQL Server's ring buffer has a maximum of 1000 events per query, 
+        ## so values above 1000 will still be capped at 1000 by the database engine. Defaults to 1000.
+        #
+        # query_completions: {}
+
+        ## @param query_errors - mapping - optional
+        ## Configure the collection of query errors from the `datadog_query_errors` XE session.
+        ##
+        ## Set `query_errors.enabled` to `true` to enable the collection of query error events.
+        ##
+        ## Use `query_errors.collection_interval` to set the interval (in seconds) for the collection of 
+        ## query error events. Defaults to 10 seconds. If you intend on updating this value,
+        ## it is strongly recommended to use a consistent value throughout all SQL Server agent deployments.
+        ##
+        ## Use `query_errors.max_events` to set the maximum number of query error events to process 
+        ## per collection. Note that SQL Server's ring buffer has a maximum of 1000 events per query, 
+        ## so values above 1000 will still be capped at 1000 by the database engine. Defaults to 1000.
+        #
+        # query_errors: {}
+
     ## Configure the collection of deadlock data.
     #
     # deadlocks_collection:

sqlserver/datadog_checks/sqlserver/sqlserver.py

@@ -53,6 +53,7 @@
 from datadog_checks.sqlserver.statements import SqlserverStatementMetrics
 from datadog_checks.sqlserver.stored_procedures import SqlserverProcedureMetrics
 from datadog_checks.sqlserver.utils import Database, construct_use_statement, parse_sqlserver_major_version
+from datadog_checks.sqlserver.xe_collection.registry import get_xe_session_handlers
 
 try:
     import datadog_agent
@@ -157,6 +158,9 @@ def __init__(self, name, init_config, instances):
         self.agent_history = SqlserverAgentHistory(self, self._config)
         self.deadlocks = Deadlocks(self, self._config)
 
+        # XE Session Handlers
+        self.xe_session_handlers = []
+
         # _database_instance_emitted: limit the collection and transmission of the database instance metadata
         self._database_instance_emitted = TTLCache(
             maxsize=1,
@@ -169,6 +173,7 @@ def __init__(self, name, init_config, instances):
         self.check_initializations.append(self.load_static_information)
         self.check_initializations.append(self.config_checks)
         self.check_initializations.append(self.make_metric_list_to_collect)
+        self.check_initializations.append(self.initialize_xe_session_handlers)
 
         # Query declarations
         self._query_manager = None
@@ -177,6 +182,13 @@ def __init__(self, name, init_config, instances):
 
         self._schemas = Schemas(self, self._config)
 
+    def initialize_xe_session_handlers(self):
+        """Initialize the XE session handlers without starting them"""
+        # Initialize XE session handlers if not already initialized
+        if not self.xe_session_handlers:
+            self.xe_session_handlers = get_xe_session_handlers(self, self._config)
+            self.log.debug("Initialized %d XE session handlers", len(self.xe_session_handlers))
+
     def cancel(self):
         self.statement_metrics.cancel()
         self.procedure_metrics.cancel()
@@ -185,6 +197,13 @@ def cancel(self):
         self._schemas.cancel()
         self.deadlocks.cancel()
 
+        # Cancel all XE session handlers
+        for handler in self.xe_session_handlers:
+            try:
+                handler.cancel()
+            except Exception as e:
+                self.log.error("Error canceling XE session handler for %s: %s", handler.session_name, e)
+
     def config_checks(self):
         if self._config.autodiscovery and self.instance.get("database"):
             self.log.warning(
@@ -810,6 +829,13 @@ def check(self, _):
                 self.sql_metadata.run_job_loop(self.tags)
                 self._schemas.run_job_loop(self.tags)
                 self.deadlocks.run_job_loop(self.tags)
+
+                # Run XE session handlers
+                for handler in self.xe_session_handlers:
+                    try:
+                        handler.run_job_loop(self.tags)
+                    except Exception as e:
+                        self.log.error("Error running XE session handler for %s: %s", handler.session_name, e)
         else:
             self.log.debug("Skipping check")
 

sqlserver/datadog_checks/sqlserver/xe_collection/__init__.py

@@ -0,0 +1,3 @@
+# (C) Datadog, Inc. 2025-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)