Hazelcast integration - Add queues, topics and reltopics and name tag. (#17367)

* Update metrics.yaml to show name of object(s) and include include topics, reltopic and queue metrics.

The hazelcast integration is not showing important details such as the map name that's being monitored. It would be great to include that, as well as include metrics for other things such as queues and topics.

* more changes

* change log.

* update order.

* changelog changes

* more change log changes

* sort metadata.csv

* sort

* sort instead

* resort this.

* Update hazelcast/metadata.csv

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update hazelcast/metadata.csv

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update hazelcast/metadata.csv

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update hazelcast/metadata.csv

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update hazelcast/metadata.csv

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* update test version to LTS hazelcast version and create queue, topic and reltopics to test out of the box.

* resolve typo

* decouple versions of mancenter / hz

* adjust dockerfile to reflect that log4j2 is included in hz 5.x

* WIP - trying to figure out how this USED to work.

* various changes to update mancenter check to be more correct.

* update metrics reported in e2e test

* add hazelcast dep

* freeze the hazelcast dep

* validate and sync licenses

* validate and sync licenses again

* updates licenses manually

* whitespace

* sort and validate metadata.csv

* add back support for hazelcast 4

---------

Co-authored-by: Brett Larson <brett.larson@chicagotrading.com>
Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>
Co-authored-by: IainMethe <20765808+IainMethe@users.noreply.github.com>

Author: 4 people

LICENSE-3rdparty.csv

@@ -35,6 +35,7 @@ flup-py3,Vendor,BSD-3-Clause,"Copyright (c) 2005, 2006 Allan Saddi <allan@saddi.
 foundationdb,PyPI,Apache-2.0,Copyright 2017 FoundationDB
 futures,PyPI,PSF,Copyright (c) 2015 Brian Quinlan
 gearman,PyPI,Apache-2.0,Copyright 2010 Yelp
+hazelcast-python-client,PyPI,Apache-2.0,"Copyright (c) 2008-2023, Hazelcast, Inc. All Rights Reserved."
 importlib-metadata,PyPI,Apache-2.0,"Copyright 2017-2019 Jason R. Coombs, Barry Warsaw"
 in-toto,PyPI,Apache-2.0,Copyright 2018 New York University
 ipaddress,PyPI,PSF,Copyright (c) 2013 Philipp Hagemeister

agent_requirements.in

@@ -29,6 +29,7 @@ enum34==1.1.10; python_version < '3.0'
 foundationdb==6.3.24; python_version > '3.0'
 futures==3.4.0; python_version < '3.0'
 gearman==2.0.2; sys_platform != 'win32' and python_version < '3.0'
+hazelcast-python-client==5.3.0; python_version > '3.0'
 importlib-metadata==2.1.3; python_version < '3.8'
 in-toto==2.0.0; python_version > '3.0'
 ipaddress==1.0.23; python_version < '3.0'

hazelcast/assets/configuration/spec.yaml

@@ -23,11 +23,8 @@ files:
       value:
         type: object
         example:
-          Active: OK
-          No Migration: WARNING
-          Frozen: CRITICAL
-          Passive: CRITICAL
-          In Transition: WARNING
+          UP: OK
+          ACTIVE: OK
     - template: instances/jmx
       overrides:
         host.required: false

hazelcast/changelog.d/17367.added

@@ -0,0 +1 @@
+Added metrics for queues, topics and reliable topics.  

hazelcast/changelog.d/17367.changed

@@ -0,0 +1,3 @@
+Drop name from excluded tags.
+Updated Hazelcast and Management Center to use version 5.x.
+Updated the status to reflect health of Management Center (Not Hazelcast cluster state).

hazelcast/datadog_checks/hazelcast/check.py

@@ -36,15 +36,31 @@ def process_mc_health_check(self):
         tags.extend(self._tags)
 
         try:
-            response = self.http.get(self._mc_health_check_endpoint)
-            response.raise_for_status()
-            status = response.json()
+            response_wrapper = self.http.get(self._mc_health_check_endpoint)
+            response_wrapper.raise_for_status()
+            response = response_wrapper.json()
         except Exception:
             self.service_check(self.SERVICE_CHECK_CONNECT, AgentCheck.CRITICAL, tags=tags)
             raise
         else:
             self.service_check(self.SERVICE_CHECK_CONNECT, AgentCheck.OK, tags=tags)
 
-        self.service_check(
-            self.SERVICE_CHECK_MC_CLUSTER_STATE, self._mc_cluster_states.get(status['managementCenterState']), tags=tags
-        )
+        status = None
+        # Hazelcast 4 and 5 have different responses to this healthcheck endpoint
+        if "status" in response:
+            # hazelcast 5:
+            # I cannot find documentation on this endpoint for management center 5.3 but it
+            # does not represent "cluster status" because that is documented and available at a
+            # different route:
+            # https://docs.hazelcast.com/management-center/5.3/integrate/cluster-metrics#operation/getAllClustersStatus
+            # It is probably the same as in 4.0 and just represents whether the management center
+            # is available.
+            status = response["status"]
+        elif "managementCenterState" in response:
+            # hazelcast 4:
+            # https://docs.hazelcast.org/docs/management-center/4.0.1/manual/html/index.html#enabling-health-check-endpoint
+            # "This endpoint responds with 200 OK HTTP status code once the Management Center web
+            # application has started"
+            status = response["managementCenterState"]
+
+        self.service_check(self.SERVICE_CHECK_MC_CLUSTER_STATE, self._mc_cluster_states.get(status), tags=tags)

hazelcast/datadog_checks/hazelcast/data/conf.yaml.example

@@ -106,11 +106,8 @@ instances:
     ## Override cluster state to `hazelcast.mc_cluster_state` service check status mapping.
     #
     # mc_cluster_states:
-    #   Active: OK
-    #   No Migration: WARNING
-    #   Frozen: CRITICAL
-    #   Passive: CRITICAL
-    #   In Transition: WARNING
+    #   UP: OK
+    #   ACTIVE: OK
 
     ## @param user - string - optional
     ## User to use when connecting to JMX.

hazelcast/datadog_checks/hazelcast/data/metrics.yaml

@@ -4,7 +4,6 @@ jmx_metrics:
   - include:
       domain: ManagementCenter
       exclude_tags:
-        - name
         - type
       attribute:
         LicenseExpirationTime:
@@ -18,7 +17,6 @@ jmx_metrics:
       type: HazelcastInstance
       exclude_tags:
         - instance
-        - name
         - type
       tags:
         hazelcast_instance: $instance
@@ -32,12 +30,14 @@ jmx_metrics:
         memberCount:
           alias: hazelcast.instance.member_count
           metric_type: gauge
+        clusterTime:
+          alias: hazelcast.instance.cluster_time
+          metric_type: gauge
   - include:
       domain: com.hazelcast
       type: HazelcastInstance.PartitionServiceMBean
       exclude_tags:
         - instance
-        - name
         - type
       tags:
         hazelcast_instance: $instance
@@ -59,7 +59,6 @@ jmx_metrics:
       type: HazelcastInstance.ManagedExecutorService
       exclude_tags:
         - instance
-        - name
         - type
       tags:
         hazelcast_instance: $instance
@@ -92,7 +91,6 @@ jmx_metrics:
       type: IMap
       exclude_tags:
         - instance
-        - name
         - type
       tags:
         hazelcast_instance: $instance
@@ -179,7 +177,6 @@ jmx_metrics:
       type: MultiMap
       exclude_tags:
         - instance
-        - name
         - type
       tags:
         hazelcast_instance: $instance
@@ -260,7 +257,6 @@ jmx_metrics:
       type: ReplicatedMap
       exclude_tags:
         - instance
-        - name
         - type
       tags:
         hazelcast_instance: $instance
@@ -326,7 +322,6 @@ jmx_metrics:
       type: Metrics
       exclude_tags:
         - instance
-        - name
         - type
         - tag.*
       tags:
@@ -437,9 +432,6 @@ jmx_metrics:
         connectionListenerCount:
           alias: hazelcast.member.connection_listener_count
           metric_type: gauge
-        connectionType:
-          alias: hazelcast.member.connection_type
-          metric_type: gauge
         count:
           alias: hazelcast.member.count
           metric_type: gauge
@@ -548,15 +540,6 @@ jmx_metrics:
         invocationTimeoutMillis:
           alias: hazelcast.member.invocation_timeout_millis
           metric_type: gauge
-        invocations.lastCallId:
-          alias: hazelcast.member.invocations.last_call_id
-          metric_type: gauge
-        invocations.pending:
-          alias: hazelcast.member.invocations.pending
-          metric_type: gauge
-        invocations.usedPercentage:
-          alias: hazelcast.member.invocations.used_percentage
-          metric_type: gauge
         ioThreadId:
           alias: hazelcast.member.io_thread_id
           metric_type: gauge
@@ -629,9 +612,6 @@ jmx_metrics:
         missingMembers:
           alias: hazelcast.member.missing_members
           metric_type: gauge
-        monitorCount:
-          alias: hazelcast.member.monitor_count
-          metric_type: gauge
         nodes:
           alias: hazelcast.member.nodes
           metric_type: gauge
@@ -659,12 +639,6 @@ jmx_metrics:
         ownerId:
           alias: hazelcast.member.owner_id
           metric_type: gauge
-        packetsReceived:
-          alias: hazelcast.member.packets_received
-          metric_type: gauge
-        packetsSend:
-          alias: hazelcast.member.packets_send
-          metric_type: gauge
         parkQueueCount:
           alias: hazelcast.member.park_queue_count
           metric_type: gauge
@@ -785,9 +759,6 @@ jmx_metrics:
         startedMigrations:
           alias: hazelcast.member.started_migrations
           metric_type: gauge
-        stateVersion:
-          alias: hazelcast.member.state_version
-          metric_type: gauge
         syncDeliveryFailureCount:
           alias: hazelcast.member.sync_delivery_failure_count
           metric_type: gauge
@@ -854,9 +825,6 @@ jmx_metrics:
         unknownTime:
           alias: hazelcast.member.unknown_time
           metric_type: gauge
-        unknownCount:
-          alias: hazelcast.member.unknown_count
-          metric_type: gauge
         unloadedClassesCount:
           alias: hazelcast.member.unloaded_classes_count
           metric_type: gauge
@@ -881,3 +849,131 @@ jmx_metrics:
         writeQueueSize:
           alias: hazelcast.member.write_queue_size
           metric_type: gauge
+
+        # metrics in 4 and 5 but at different locations
+        # hazelcast 5
+        lastCallId:
+          alias: hazelcast.member.invocations.last_call_id
+          metric_type: gauge
+        pending:
+          alias: hazelcast.member.invocations.pending
+          metric_type: gauge
+        usedPercentage:
+          alias: hazelcast.member.invocations.used_percentage
+          metric_type: gauge
+        unknownCount:
+          alias: hazelcast.member.unknown_count
+          metric_type: gauge
+        # hazelcast 4
+        invocations.lastCallId:
+          alias: hazelcast.member.invocations.last_call_id
+          metric_type: gauge
+        invocations.pending:
+          alias: hazelcast.member.invocations.pending
+          metric_type: gauge
+        invocations.usedPercentage:
+          alias: hazelcast.member.invocations.used_percentage
+          metric_type: gauge
+        unknowsnCount: # typo on hazelcast's side
+          alias: hazelcast.member.unknown_count
+          metric_type: gauge
+
+        # hazelcast 4 only
+        connectionType:
+          alias: hazelcast.member.connection_type
+          metric_type: gauge
+        monitorCount:
+          alias: hazelcast.member.monitor_count
+          metric_type: gauge
+        packetsReceived:
+          alias: hazelcast.member.packets_received
+          metric_type: gauge
+        packetsSend:
+          alias: hazelcast.member.packets_send
+          metric_type: gauge
+        stateVersion:
+          alias: hazelcast.member.state_version
+          metric_type: gauge
+  # Queue
+  - include:
+      domain: com.hazelcast
+      type: IQueue
+      exclude_tags:
+        - instance
+        - type
+      tags:
+        hazelcast_instance: $instance
+      attribute:
+        localMinAge:
+          alias: hazelcast.iqueue.minimum_age
+          metric_type: gauge
+        localMaxAge:
+          alias: hazelcast.iqueue.maximum_age
+          metric_type: gauge
+        localAverageAge:
+          alias: hazelcast.iqueue.average_age
+          metric_type: gauge
+        localOwnedItemCount:
+          alias: hazelcast.iqueue.owned_item_count
+          metric_type: gauge
+        localBackupItemCount:
+          alias: hazelcast.iqueue.backup_item_count
+          metric_type: gauge
+        localOfferOperationCount:
+          alias: hazelcast.iqueue.offer_operation_count
+          metric_type: gauge
+        localRejectedOfferOperationCount:
+          alias: hazelcast.iqueue.rejected_offer_operation_count
+          metric_type: gauge
+        localPollOperationCount:
+          alias: hazelcast.iqueue.poll_operation_count
+          metric_type: gauge
+        localEmptyPollOperationCount:
+          alias: hazelcast.iqueue.empty_poll_operation_count
+          metric_type: gauge
+        localOtherOperationsCount:
+          alias: hazelcast.iqueue.other_operation_count 
+          metric_type: gauge
+        localEventOperationCount:
+          alias: hazelcast.iqueue.event_operation_count
+          metric_type: gauge
+
+  # ReliableTopic
+  - include:
+      domain: com.hazelcast
+      type: ReliableTopic
+      exclude_tags:
+        - instance
+        - type
+      tags:
+        hazelcast_instance: $instance
+      attribute:
+        localCreationTime:
+          alias: hazelcast.reliabletopic.creation_time
+          metric_type: gauge
+        localPublishOperationCount:
+          alias: hazelcast.reliabletopic.publish_operation_count
+          metric_type: gauge
+        localReceiveOperationCount:
+          alias: hazelcast.reliabletopic.receive_operation_count
+          metric_type: gauge
+
+  # Topic
+  - include:
+      domain: com.hazelcast
+      type: ITopic
+      exclude_tags:
+        - instance
+        - type
+      tags:
+        hazelcast_instance: $instance
+      attribute:
+        localCreationTime:
+          alias: hazelcast.topic.creation_time
+          metric_type: gauge
+        localPublishOperationCount:
+          alias: hazelcast.topic.publish_operation_count
+          metric_type: gauge
+        localReceiveOperationCount:
+          alias: hazelcast.topic.receive_operation_count
+          metric_type: gauge

hazelcast/datadog_checks/hazelcast/utils.py

@@ -3,13 +3,12 @@
 # Licensed under a 3-clause BSD style license (see LICENSE)
 from datadog_checks.base.constants import ServiceCheck
 
-# https://docs.hazelcast.org/docs/management-center/latest/manual/html/index.html#cluster-state
+# Cluster state is not reflected here, this check is purely for Management Center.
 MC_CLUSTER_STATES = {
-    'Active': ServiceCheck.OK,
-    'No Migration': ServiceCheck.WARNING,
-    'Frozen': ServiceCheck.CRITICAL,
-    'Passive': ServiceCheck.CRITICAL,
-    'In Transition': ServiceCheck.WARNING,
+    # hazelcast 5:
+    'UP': ServiceCheck.OK,
+    # hazelcast 4:
+    'ACTIVE': ServiceCheck.OK,
 }
 
 

hazelcast/hatch.toml

@@ -2,9 +2,14 @@
 
 [[envs.default.matrix]]
 python = ["3.11"]
-version = ["4.0"]
+version = ["4.0", "5.0"]
 
 [envs.default.overrides]
 matrix.version.env-vars = [
   { key = "HAZELCAST_VERSION", value = "4.0.1", if = ["4.0"] },
+  { key = "HAZELCAST_VERSION", value = "5.3.7", if = ["5.0"] },
+  { key = "HAZELCAST_MANCENTER_VERSION", value = "4.0.1", if = ["4.0"] },
+  { key = "HAZELCAST_MANCENTER_VERSION", value = "5.3.3", if = ["5.0"] },
+  { key = "HAZELCAST_MC_INIT_CMD", value = "./mc-conf.sh", if = ["4.0"] },
+  { key = "HAZELCAST_MC_INIT_CMD", value = "./bin/mc-conf.sh", if = ["5.0"] },
 ]