internal-metrics.md

internal-metrics

Emits metrics about the internal state of the agent. Useful for debugging performance issues with the agent and to ensure the agent isn't overloaded.

This can also scrape any HTTP endpoint that exposes metrics as a JSON array containing JSON-formatted SignalFx datapoint objects. It is roughly analogous to the prometheus-exporter monitor except for SignalFx datapoints.

monitors:
  - type: internal-metrics

Monitor Type: internal-metrics

Monitor Source Code

Accepts Endpoints: Yes

Multiple Instances Allowed: Yes

Configuration

Config option	Required	Type	Description
host	no	string	Defaults to the top-level internalStatusHost option
port	no	integer	Defaults to the top-level internalStatusPort option
path	no	string	The HTTP request path to use to retrieve the metrics (default: /metrics)

Metrics

The following table lists the metrics available for this monitor. Metrics that are marked as Included are standard metrics and are monitored by default.

Name	Type	Included	Description
sfxagent.active_monitors	gauge		The total number of monitor instances actively working
sfxagent.active_observers	gauge		The number of observers configured and running
sfxagent.configured_monitors	gauge		The total number of monitor configurations
sfxagent.datapoint_channel_len	gauge		The total number of datapoints that have been emitted by monitors but have yet to be accepted by the writer. This number should be 0 most of the time. This will max out at 3000, at which point no datapoints will be generated by monitors. If it does max out, it indicates a bug or extreme CPU starvation of the agent.
sfxagent.datapoint_requests_active	gauge		The total number of outstanding requests to ingest currently active. If this is consistently hovering around the writer.maxRequests setting, that setting should probably be increased to give the agent more bandwidth to send datapoints.
sfxagent.datapoints_filtered	cumulative		The total number of datapoints that were filtered out in the writer. This does not include datapoints filtered by monitor-specific filters.
sfxagent.datapoints_in_flight	gauge		The total number of datapoints that have been sent out in a request to ingest but have yet to receive confirmation from ingest that they have been received (i.e. the HTTP response hasn't been gotten).
sfxagent.datapoints_received	cumulative		The total number of non-filtered datapoints received by the agent writer since it last started. This number should generally equal sfxagent.datapoints_sent + sfxagent.datapoints_waiting + sfxagent.datapoints_in_flight, although sampling timing issues might cause it to temporarily not be.
sfxagent.datapoints_sent	cumulative		The total number of datapoints sent by the agent writer since it last started
sfxagent.datapoints_waiting	gauge		The total number of datapoints that have been accepted by the writer but have yet to be sent out to ingest over HTTP. If this continues to grow it indicates that datapoints are not being sent out fast enough and the writer.maxRequests setting should be increased.
sfxagent.dim_request_senders	gauge		Current number of worker goroutines active that can send dimension updates.
sfxagent.dim_updates_completed	cumulative		Total number of dimension property updates successfully completed
sfxagent.dim_updates_currently_delayed	gauge		Current number of dimension updates that are being delayed to avoid sending spurious updates due to flappy dimension property sets.
sfxagent.dim_updates_dropped	cumulative		Total number of dimension property updates that were dropped, due to an overfull buffer of dimension updates pending.
sfxagent.dim_updates_failed	cumulative		Total number of dimension property updates that failed for some reason. The failures should be logged.
sfxagent.dim_updates_flappy_total	cumulative		Total number of dimension property updates that ended up replacing a dimension property set that was being delayed.
sfxagent.dim_updates_started	cumulative		Total number of dimension property updates requests started, but not necessarily completed or failed.
sfxagent.discovered_endpoints	gauge		The number of discovered service endpoints. This includes endpoints that do not have any matching monitor configuration discovery rule.
sfxagent.events_buffered	gauge		The total number of events that have been emitted by monitors but have yet to be sent to SignalFx
sfxagent.events_sent	cumulative		The total number of events sent by the agent since it last started
sfxagent.go_frees	cumulative		Total number of heap objects freed throughout the lifetime of the agent
sfxagent.go_heap_alloc	gauge		Bytes of live heap memory (memory that has been allocated but not freed)
sfxagent.go_heap_idle	gauge		Bytes of memory that consist of idle spans (that is, completely empty spans of memory)
sfxagent.go_heap_inuse	gauge		Size in bytes of in use spans
sfxagent.go_heap_released	gauge		Bytes of memory that have been returned to the OS. This is quite often 0. sfxagent.go_heap_idle - sfxagent.go_heap_release is the memory that Go is retaining for future heap allocations.
sfxagent.go_heap_sys	gauge		Virtual memory size in bytes of the agent. This will generally reflect the largest heap size the agent has ever had in its lifetime.
sfxagent.go_mallocs	cumulative		Total number of heap objects allocated throughout the lifetime of the agent
sfxagent.go_next_gc	gauge		The target heap size -- GC tries to keep the heap smaller than this
sfxagent.go_num_gc	gauge		The number of GC cycles that have happened in the agent since it started
sfxagent.go_stack_inuse	gauge		Size in bytes of spans that have at least one goroutine stack in them
sfxagent.go_total_alloc	cumulative		Total number of bytes allocated to the heap throughout the lifetime of the agent
sfxgent.go_num_goroutine	gauge		Number of goroutines in the agent