Fix CouchDB 2+ message_queues metric collection (#19521)

* Fix CouchDB 2+ message_queues metric collection

At the moment, the `count` value in the metric block is recorded
as a gauge. That value is the number of instances of each type
of message queue. Say you have two database files open, then
`message_queues.couch_file.count` will be `2`. That information
is already available in the `_stats` endpoint as `open_os_files`.

The actually valuable part of the message_queue metric is the
`max` entry, it shows that the longest backlog of messages for
each instance type. A longer backlog means a reduction in
performance. Without this patch, this value is not available.

Technically all of the `max`, `min`, `50`, `90`, `99` are
valuable, but I wanted to keep this patch as small as possible
until it is clear the change will be accepted.

I’m happy to add the other fields as well if desired.

* OPTIONAL: expose more useful message_queues metrics

* lint

* fix: add assertions for new stats

* add new message queue metric descriptions

* changelog

* move changelog

* delete

---------

Co-authored-by: steveny91 <steven.yuen@datadoghq.com>

Author: janl

couch/changelog.d/19521.added

@@ -0,0 +1 @@
+Add support for collecting different metric aggregations for message_queue metrics.

couch/datadog_checks/couch/couch.py

@@ -247,12 +247,18 @@ def _build_system_metrics(self, data, tags, prefix='couchdb.erlang'):
                     queue_tags = list(tags)
                     queue_tags.append("queue:{0}".format(queue))
                     if isinstance(val, dict):
-                        if 'count' in val:
-                            self.gauge("{0}.{1}.size".format(prefix, key), val['count'], queue_tags)
+                        if 'max' in val:
+                            self.gauge("{0}.{1}.max".format(prefix, key), val['max'], queue_tags)
+                        if 'min' in val:
+                            self.gauge("{0}.{1}.min".format(prefix, key), val['min'], queue_tags)
+                        if '50' in val:
+                            self.gauge("{0}.{1}.50".format(prefix, key), val['50'], queue_tags)
+                        if '90' in val:
+                            self.gauge("{0}.{1}.90".format(prefix, key), val['90'], queue_tags)
+                        if '99' in val:
+                            self.gauge("{0}.{1}.99".format(prefix, key), val['99'], queue_tags)
                         else:
-                            self.agent_check.log.debug(
-                                "Queue %s does not have a key 'count'. It will be ignored.", queue
-                            )
+                            self.agent_check.log.debug("Queue %s does not have any keys. It will be ignored.", queue)
                     else:
                         self.gauge("{0}.{1}.size".format(prefix, key), val, queue_tags)
             elif key == "distribution":

couch/metadata.csv

@@ -260,7 +260,12 @@ couchdb.erlang.memory.ets,gauge,,byte,,,0,couchdb,,
 couchdb.erlang.memory.other,gauge,,byte,,,0,couchdb,,
 couchdb.erlang.memory.processes,gauge,,byte,,,0,couchdb,,
 couchdb.erlang.memory.processes_used,gauge,,byte,,,0,couchdb,,
-couchdb.erlang.message_queues.size,gauge,,,,,0,couchdb,,
+couchdb.erlang.message_queues.size,gauge,,,,number of message queues,0,couchdb,,
+couchdb.erlang.message_queues.max,gauge,,,,max message queue length,0,couchdb,,
+couchdb.erlang.message_queues.min,gauge,,,,min message queue length,0,couchdb,,
+couchdb.erlang.message_queues.50,gauge,,,,50th percentile message queue length,0,couchdb,,
+couchdb.erlang.message_queues.90,gauge,,,,90th percentile message queue length,0,couchdb,,
+couchdb.erlang.message_queues.99,gauge,,,,99th percentile message queue length,0,couchdb,,
 couchdb.erlang.os_proc_count,gauge,,,,,0,couchdb,,
 couchdb.erlang.process_count,gauge,,,,,0,couchdb,,
 couchdb.erlang.process_limit,gauge,,,,,0,couchdb,,