Skip to content

Latest commit

 

History

History
70 lines (42 loc) · 4.05 KB

metrics.md

File metadata and controls

70 lines (42 loc) · 4.05 KB
layout title permalink redirect_from
post
METRICS
/docs/metrics
/metrics.md/
/docs/metrics.md/

UPDATE: to build aisnode with support for StatsD, use the namesake build tag: statsd. Note that Prometheus is the default supported option.

See also: conditional linkage

UPDATE: for complete reference that includes all supported metric names (both internal and visible externally), their respective types, descriptions, and labels, please refer to:

As far Prometheus, one immediate (low-level) way to view the metrics in action would be curl:

$ curl http://<aistore-node-ip-or-hostname>:<port>/metrics

## or, possibly:

$ curl https://<aistore-node-ip-or-hostname>:<port>/metrics

UPDATE: rest of this document may contain a slightly outdated information.

===================

AIStore tracks, logs, and reports a large and growing number of counters, latencies and throughputs including (but not limited to) metrics that reflect cluster recovery and global rebalancing, all extended long-running operations, and, of course, the basic read, write, list transactions and more.

A somewhat outdated example of how these metrics show up in the Grafana dashboard follows:

AIS loader metrics

But generally, complete observability is supported with a variety of tools, including:

  1. System logs
  2. CLI and, in particular, ais show performance command
  3. Prometheus
  4. Any StatsD compliant backend including Graphite/Grafana

AIStore includes aisloader - a powerful tool that we use to simulate a variety of AI workloads. For numerous command-line options and usage examples, please see aisloader and How To Benchmark AIStore.

Or, just run make aisloader; aisloader and see its detailed online help. Note as well that aisloader is fully StatsD-enabled and supports detailed protocol-level tracing with runtime on and off switching.

StatsD and Prometheus

AIStore generates a growing number of detailed performance metrics. Other than AIS logs, the stats can be viewed via:

  • StatsD/Grafana visualization or
  • Prometheus visualization

StatsD publishes local statistics to a compliant backend service (e.g., Graphite) for easy and powerful stats aggregation and visualization.

AIStore is a fully compliant Prometheus exporter that natively supports Prometheus stats collection. There's no special configuration - the only thing required to enable the corresponding integration is letting AIStore know whether to publish its stats via StatsD or Prometheus.

The StatsD/Grafana option imposes a certain easy-to-meet requirement on the AIStore deployment. Namely, it requires that StatsD daemon (aka service) is deployed locally with each AIS target and with each AIS proxy.

At startup AIStore daemons, both targets and gateways, try to UDP-ping their respective local StatsD daemons on the UDP port 8125 unless redefined via environment AIS_STATSD_PORT. You can disable StatsD reachability probing by setting another environment variable - AIS_STATSD_PROBE - to false or no.

If StatsD server is not listening on the local 8125, the local AIS target (or proxy) will then run without StatsD, and the corresponding stats won't be captured and won't be visualized.

For details on all StatsD-supported backends, please refer to this document.

For Prometheus integration, please refer to this separate document