How are you doing metrics?

[gthomson31]

Hi All

I have since deployed ARC and have managed to get the self hosted runners up and running.
Just looking for some advice on how best to monitor and maintain this deployment ?

We currently use Grafana internally so looking to see how I could use this with the runner deployment metrics

Any help appreciated!

[debugger24]

You can use Prometheus to scrape metrics on action-controller-runner.

Then you can connect Prometheus to Grafana

The current version exposed following metrics.

• horizontalrunnerautoscaler_spec_min_replicas
• horizontalrunnerautoscaler_spec_max_replicas
• horizontalrunnerautoscaler_status_desired_replicas

In next release we will have following metrics too apart from above ones (PR already merged)

• horizontalrunnerautoscaler_replicas_desired
• horizontalrunnerautoscaler_runners
• horizontalrunnerautoscaler_runners_registered
• horizontalrunnerautoscaler_runners_busy
• horizontalrunnerautoscaler_terminating_busy
• horizontalrunnerautoscaler_necessary_replicas
• horizontalrunnerautoscaler_workflow_runs_completed
• horizontalrunnerautoscaler_workflow_runs_in_progress
• horizontalrunnerautoscaler_workflow_runs_queued
• horizontalrunnerautoscaler_workflow_runs_unknown

For some constraints we were using only the PercentBusy metric and not webhook metrics, we created a Grafana dashboard to track autoscaling metrics.

[nitinses]

@debugger24 Could you share the grafana json please

[remover]

• horizontalrunnerautoscaler_replicas_desired
• horizontalrunnerautoscaler_runners
• horizontalrunnerautoscaler_runners_registered
• horizontalrunnerautoscaler_runners_busy
• horizontalrunnerautoscaler_terminating_busy
• horizontalrunnerautoscaler_necessary_replicas
• horizontalrunnerautoscaler_workflow_runs_completed
• horizontalrunnerautoscaler_workflow_runs_in_progress
• horizontalrunnerautoscaler_workflow_runs_queued
• horizontalrunnerautoscaler_workflow_runs_unknown

are these metrics usable if you're doing webhook based autoscaling, i.e. workflow_job?

[balajisa09]

@debugger24 can you provide the grafana dashboard json ?

[gcosta-lively]

If you could share the grafana dashboard json it would be very helpful , as it saves me half day 🫶

[alopezsanchez]

Hey, I just published this Grafana dashboard.
Any suggestion is welcome!

[gthomson31]

@debugger24 that's great, thanks for the help!
I was playing around with metrics yesterday and managed to get Prometheus deployed in our cluster but couldnt get the ARC ServiceMonitor to register with Prometheus.

set {
name = "metrics.serviceMonitor"
value = true
}

Did you have to do any additional Config to get this working?
Particularly around RBAC and Kube Proxy

Installed Prometheus Stack :
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

[debugger24]

couldnt get the ARC ServiceMonitor to register with Prometheus.

Can you share more details what errors are you getting ?

[gthomson31]

Not Seeing any specific error message just not seeing it picked up under service discovery - however github-actions-exporter is being picked up also using serviceMonitor.

[gthomson31]

Actions Metrics Yaml Output

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: github-actions-runner-controller
    meta.helm.sh/release-namespace: actions-runner-system
  creationTimestamp: "2022-08-15T13:39:13Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: github-actions-runner-controller
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: actions-runner-controller
    app.kubernetes.io/version: 0.25.2
    helm.sh/chart: actions-runner-controller-0.20.2
  name: github-actions-runner-controller-service-monitor
  namespace: actions-runner-system
  resourceVersion: "31879047"
  uid: 2d178124-b1f4-4790-a8f0-49389f8d819f
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    path: /metrics
    port: metrics-port
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/instance: github-actions-runner-controller
      app.kubernetes.io/name: actions-runner-controller

---

Github Actions Exporter Yaml Output

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: eks-monitoring-github-actions-exporter
    meta.helm.sh/release-namespace: actions-runner-system
  creationTimestamp: "2022-08-16T13:33:41Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    release: prometheus
  name: eks-monitoring-github-actions-exporter
  namespace: actions-runner-system
  resourceVersion: "32283244"
  uid: ccc110f9-0975-458c-b75b-98123c322f6a
spec:
  endpoints:
  - interval: 300s
    path: /metrics
    targetPort: 9999
  jobLabel: eks-monitoring-github-actions-exporter
  namespaceSelector:
    matchNames:
    - actions-runner-system
  selector:
    matchLabels:
      app: github-actions-exporter
      release: eks-monitoring-github-actions-exporter

[gthomson31]

Tried to Disable the Proxy but leave ServiceMonitor enabled - Still not being picked up

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: github-actions-runner-controller
    meta.helm.sh/release-namespace: actions-runner-system
  creationTimestamp: "2022-08-16T14:07:40Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: github-actions-runner-controller
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: actions-runner-controller
    app.kubernetes.io/version: 0.25.2
    helm.sh/chart: actions-runner-controller-0.20.2
  name: github-actions-runner-controller-service-monitor
  namespace: actions-runner-system
  resourceVersion: "32293595"
  uid: 8af8fc5e-d954-4730-a9f8-4c207eb8237e
spec:
  endpoints:
  - path: /metrics
    port: metrics-port
  selector:
    matchLabels:
      app.kubernetes.io/instance: github-actions-runner-controller
      app.kubernetes.io/name: actions-runner-controller

[debugger24]

In my case,

1. We were not using Prometheus Operator
2. We don't need RBAC as of now because we have complete cluster dedicated to gh runner.

Config I used was

In https://prometheus-community.github.io/helm-charts/

 extraScrapeConfigs: |
  - job_name: 'action-runner-controller'
    scheme: http
    metrics_path: /metrics
    static_configs:
    - targets:
      - my-github-controller-actions-runner-controll-metrics-service.actions-runner-system.svc:8443

In https://actions-runner-controller.github.io/actions-runner-controller

metrics:
  serviceAnnotations: {}
  serviceMonitor: false
  serviceMonitorLabels: {}
  port: 8443
  proxy:
    enabled: false
    image:
      repository: quay.io/brancz/kube-rbac-proxy
      tag: v0.13.0

Disabled proxy because I don't need RBAC at the moment. Disabled serviceMonitor because I am not using Prometheus Operator

If you use Prometheus Operator, and enable serviceMonitor I am assuming this helm chart will create the required RBAC permissions.

[gthomson31]

Strangely when trying to hit the endpoint I am getting Connection Refused ?

curl 10.200.20.60:8443/metrics
curl: (7) Failed to connect to 10.200.20.60 port 8443 after 37 ms: Connection refused

However when I enable port forwarding it works

kubectl port-forward svc/github-actions-runner-controller-metrics-service  5555:8443
Forwarding from 127.0.0.1:5555 -> 8443
Forwarding from [::1]:5555 -> 8443


curl 127.0.0.1:5555/metrics
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 1

[leadelngalame1611]

Please i need some help in getting all the metrics provided by ARC.
From my understanding, the following metrics

horizontalrunnerautoscaler_replicas_desired
horizontalrunnerautoscaler_runners
horizontalrunnerautoscaler_runners_registered
horizontalrunnerautoscaler_runners_busy
horizontalrunnerautoscaler_terminating_busy
horizontalrunnerautoscaler_necessary_replicas
horizontalrunnerautoscaler_workflow_runs_completed
horizontalrunnerautoscaler_workflow_runs_in_progress
horizontalrunnerautoscaler_workflow_runs_queued
horizontalrunnerautoscaler_workflow_runs_unknown

have been added to the new release ( runner-controller 0.22.0 ). So i upgraded my controllers to the new version but still just seeing only the following metrics

horizontalrunnerautoscaler_spec_min_replicas
horizontalrunnerautoscaler_spec_max_replicas
horizontalrunnerautoscaler_status_desired_replicas

We do use CDK to deploy the Helm resources in Eks and my configuration is as follows

const actionsRunnerCtrlHelmChart = new eks.HelmChart(this, "gh", {
      cluster,
      chart: "actions-runner-controller",
      repository: "https://actions-runner-controller.github.io/actions-runner-controller",
      namespace,
      createNamespace: false,
      wait: true,
      values: {
        labels: {
          service: "github-action-runner",
        },
        syncPeriod: "1m",
        defaultScaleDownDelay: "2m",
        scope: {
          watchNamespace: namespace,
          singleNamespace: true,
        },
        authSecret: {
          name: "github-runner-token",
          create: false,
          enabled: false,
        },
        env: [
          {
            name: "GITHUB_TOKEN",
            valueFrom: {
              secretKeyRef: {
                name: "github-runner-token",
                key: "github_token",
              },
            },
          },
        ],
        additionalVolumes: [
          {
            name: "github-runner-token-volume",
            csi: {
              driver: "secrets-store.csi.k8s.io",
              readOnly: true,
              volumeAttributes: {
                secretProviderClass: "github-runner-secret-provider-class",
              },
            },
          },
        ],
        additionalVolumeMounts: [
          {
            name: "github-runner-token-volume",
            mountPath: "/token",
            readOnly: true,
          },
        ],
        serviceAccount: {
          name: serviceAccountName,
          create: false,
        },
        metrics: {
          serviceMonitor: true,
          serviceMonitorLabels: {
            service: "github-action-runner",
          },
          port: 8080,
          proxy: {
            enabled: false,
          },
        },
        podAnnotations,
        tolerations,
        affinity,
        githubWebhookServer: {
          logFormat: "json",
          logLevel: "info",
          fullnameOverride: "gh-webhook",
          nameOverride: "gh-webhook",
          enabled: true,
          tolerations: [...tolerations, ...tolerations_graviton],
          affinity,
          podAnnotations,
          service: {
            type: "NodePort",
            annotations: {},
            ports: [
              {
                protocol: "TCP",
                port: 80,
                targetPort: "http",
                name: "http",
              },
            ],
          },
          ingress: {
            enabled: true,
            ingressClassName: "alb",
            annotations: {
              // "kubernetes.io/ingress.class": "alb",   ##This chart requires to set ingressClassName as above
              "alb.ingress.kubernetes.io/tags": `team=me,environment=${props!.environment},service=me`,
              "alb.ingress.kubernetes.io/scheme": "private",
              "alb.ingress.kubernetes.io/target-type": "instance",
              "cert-manager.io/cluster-issuer": "default-cluster-issuer",
              "alb.ingress.kubernetes.io/listen-ports": '[{"HTTPS":443}]',
              "alb.ingress.kubernetes.io/group.name": "private",
            },
            hosts: [
              {
                host: "example.com",
                paths: [
                  {
                    path: "/actions-runner-controller-github-webhook-server",
                    pathType: "Prefix",
                  },
                ],
              },
            ],
            tls: [
              {
                hosts: ["example.com"],
                secretName: "gh-scaler-cert",
              },
            ],
          },
        },
      },
    });

Chart-Version: actions-runner-controller-0.22.0
App-Version: 0.27.0

Please can someone help me figure this out?

[Moser-ss]

Just be aware that Helm doesn't update CRDs, if that is necessary to have the metrics you need to update manually the CRD

[rekha-prakash-maersk]

Hi @Moser-ss , can you please point me to a document on how to manually update the CRD ?

I am also using the helm chart and only seeing half of the metrics coming up.

[Moser-ss]

The CDRs are just normal Kubernetes resources so you can use kubectl apply and update CDR . The CDRs are located here charts/actions-runner-controller/crds
Basically you can do `kubectl apply -f

Also the metrics server works with the input of the webhooks

[rekha-prakash-maersk]

Hi @Moser-ss Thank you for your reply.

I have the metrics annotation as mentioned in the troubleshooting document, and the prometheus available at cluster level

spec:
replicas: 1
template:
metadata:
labels:
product: "sh-runner"
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080"

But the workflow metrics alone not coming up in the grafana, which is the metric I am really after, so I am kind of stuck. can you please help !

[rekha-prakash-maersk]

@Moser-ss , I use fluxcd for deployment , so added crds upgrade in the template itself, but still only workflow metrics are missing though. any idea what else I might be missing ? TIA!!

install:
    remediation:
      retries: -1
    crds: CreateReplace
  upgrade:
    crds: CreateReplace

[philthynz]

Running chart version 0.22.0 I also can't see any of the new metrics. I haven't upgraded, I've just added some values to my Prometheus and ARC Helm charts:

kube-prometheus:
Done this to get Prometheus operator working accross namespaces.

prometheus:
  prometheusSpec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: agentpool
              operator: In
              values:
              - "${agentPool}"
    podMonitorNamespaceSelector:
      any: true
    podMonitorSelector: {}
    podMonitorSelectorNilUsesHelmValues: false
    ruleNamespaceSelector:
      any: true
    ruleSelector: {}
    ruleSelectorNilUsesHelmValues: false
    serviceMonitorNamespaceSelector:
      any: true
    serviceMonitorSelector: {}
    serviceMonitorSelectorNilUsesHelmValues: false

ARC:
Added additional config to create the service monitors

metrics:
  serviceAnnotations: {}
  serviceMonitor: true
  serviceMonitorLabels: {}
  port: 8443
  proxy:
    enabled: false
    image:
      repository: quay.io/brancz/kube-rbac-proxy
      tag: v0.13.1

actionsMetrics:
  serviceAnnotations: {}
  serviceMonitor: true
  serviceMonitorLabels: {}
  port: 8443
  proxy:
    enabled: false
    image:
      repository: quay.io/brancz/kube-rbac-proxy
      tag: v0.13.1

actionsMetricsServer:
  enabled: true

Pods:

NAME                                        READY   STATUS    RESTARTS        AGE
build-arc-bc4f7cfff-f7hsr                   1/1     Running   0          105s
build-arc-metrics-server-6c4c8747d-zt6g9    1/1     Running   0          33s
build-arc-webhook-server-84d44644d5-4pnd8   1/1     Running   0          26s
build-general-linux-dev-2qrk2-fgmrz         1/1     Running   0          5d21h
build-general-linux-dev-2qrk2-tbq2v         1/1     Running   0          5d22h
build-general-linux-dev-2qrk2-vjzwt         1/1     Running   0          5d21h
build-general-linux-dev-2qrk2-x6p82         1/1     Running   0          5d21h

The build-arc-bc4f7cfff-f7hsr build-arc-metrics-server-6c4c8747d-zt6g9 and build-arc-webhook-server-84d44644d5-4pnd8 pods, are running the metrics ports:

- name: metrics-port
  containerPort: 8443
  protocol: TCP

I can't see those new metrics on the build-arc-bc4f7cfff-f7hsr pod.

runner@build-general-linux-dev-2qrk2-tbq2v:/$ curl -k http://172.25.53.8:8443/metrics | grep horizontalrunnerautoscaler_
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

# HELP horizontalrunnerautoscaler_spec_max_replicas maxReplicas of HorizontalRunnerAutoscaler
# TYPE horizontalrunnerautoscaler_spec_max_replicas gauge
horizontalrunnerautoscaler_spec_max_replicas{horizontalrunnerautoscaler="build-general-linux-dev-autoscaler",namespace="actions-runner-system-build"} 8
100# HELP horizontalrunnerautoscaler_spec_min_replicas minReplicas of HorizontalRunnerAutoscaler
 6# TYPE horizontalrunnerautoscaler_spec_min_replicas gauge
587horizontalrunnerautoscaler_spec_min_replicas{horizontalrunnerautoscaler="build-general-linux-dev-autoscaler",namespace="actions-runner-system-build"} 4
5    0 65875    0     0  27.4M      0 --:--:-- --:--:-- --:--:-- 31.4M

[stelligent-joao]

I have the exact same problem. The pods are up but no metrics coming through

[fralvarop]

Also this issue is open from another user: #2799

I'm having a similar issue. I have ARC release update to the last version, CRDs are update also. But I just get 2 metrics

• horizontalrunnerautoscaler_spec_max_replicas
• horizontalrunnerautoscaler_spec_min_replicas

This is my configuration

podAnnotations:
  "prometheus.io/scrape": "true"
  "prometheus.io/path": "/metrics"
  "prometheus.io/port": "8080"

metrics:
  serviceMonitor:
    enable: true
  port: 8080
  proxy:
    enabled: false

githubWebhookServer:
  enabled: true
  ports:
  - nodePort: 33080

actionsMetrics:
  serviceMonitor:
    enable: true
    port: 8080
  proxy:
    enabled: false

[m-perrott]

This is the same for us. Everything on latest but only able to get these 2 metrics whilst using prometheus-community helm chart

[rekha-prakash-maersk]

Hi, I am using v.027.5 image version and though I have jobs in queue, I dont find see these metrics reflecting the same,

# HELP workqueue_depth Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="horizontalrunnerautoscaler-controller"} 0
workqueue_depth{name="runner-controller"} 0
workqueue_depth{name="runnerdeployment-controller"} 0
workqueue_depth{name="runnerpersistentvolume-controller"} 0
workqueue_depth{name="runnerpersistentvolumeclaim-controller"} 0
workqueue_depth{name="runnerpod-controller"} 0
workqueue_depth{name="runnerreplicaset-controller"} 0
workqueue_depth{name="runnerset-controller"} 0

I am looking for a metric on how many jobs are in queue, so I can setup alert on that, if the queue is too big.

[dhutsj]

I can also only see two 2 metrics with everything latest? Any idea?

horizontalrunnerautoscaler_spec_max_replicas
horizontalrunnerautoscaler_spec_min_replicas

[jameshounshell]

Anyone looking for a dashboard that works with the new metrics should check out this blog: https://www.kenmuse.com/blog/enabling-github-arc-metrics/

I have found that the cardinality on the gha_* metrics can be insane because they include runner_name (pod name) and runner_id (job id) essentially making one series per data point which prevents even using rate() without a subquery to discard runner_name/runner_id. Additionally the two histogram metrics have 48 buckets which is much higher than the prometheus default/recommended 10.

To clean up your metrics try using a service monitor configured like this. You'll need two, one for the controller, one for the listener (bulk of metrics). This is the one for the listener:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scaleset-metrics
  namespace: github-actions
spec:
  endpoints:
  - interval: 1m
    metricRelabelings:
      # drop all infinite cardinality labels
      - action: labeldrop
        regex: ^runner_id$
      - action: labeldrop
        regex: ^runner_name$
      # For histograms gha_job_execution_duration and gha_job_startup_duration
      # which have 48 buckets by default
      # pick your buckets to keep, must keep "+Inf" bucket
      # mind the final empty else `(...|)`, which is required for this sneaky regex to work
      - action: keep
        regex: (10|60|300|600|3600|\+Inf|)
        sourceLabels: [le]
    path: /metrics
    port: http-metrics
    scrapeTimeout: 30s
    # we use istio so this may not be necessary for you
    scheme: https
    tlsConfig:
      caFile: /etc/prom-certs/root-cert.pem
      certFile: /etc/prom-certs/cert-chain.pem
      insecureSkipVerify: true
      keyFile: /etc/prom-certs/key.pem
  selector:
    matchLabels:
      app.kubernetes.io/component: runner-scale-set-listener
      app.kubernetes.io/part-of: gha-runner-scale-set

[rekha-prakash-maersk]

@jameshounshell This is really helpful ! I will try this out.
I also have one more use-case where the listener metrics resets during its restart making the old metrics disappear. When I try to get total jobs completed or total execution times for last week for all jobs, these resets cause issue. any idea how to retain the old metrics to in Prometheus?

[rekha-prakash-maersk]

@jameshounshell can you please share the sample config for controller metrics as well ?

I am currently using below config in the controller values.yaml file,

podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080"
metrics:
controllerManagerAddr: ":8080"
listenerAddr: ":8080"
listenerEndpoint: "/metrics"

[jameshounshell]

I haven't experienced metrics "disappearing" but we've only been running it for 4 days now. I'll restart to verify what happens. Prometheus counters are supposed to be resilient against restarts by their design. Prometheus should also be persisting your data so that could be a whole other issue (possibly misconfigured).

I think you might be experiencing something similar to what I described about how the metrics have too high cardinality. Try a query (with inner subquery) like this:

sum by (job_result)(rate(
    (sum without (runner_id,runner_name)(gha_completed_jobs_total))
    [5m:]
))

Here's the metrics config we use (which should be as described in the docs):

metrics:
  controllerManagerAddr: ":8080"
  listenerAddr: ":8080"
  listenerEndpoint: "/metrics"
podAnnotations:
  "prometheus.io/scrape": "true"
  "prometheus.io/path": "/metrics"
  "prometheus.io/port": "8080"

[jameshounshell]

Whelp update. It seems like runner_id/runner_name is a known issue and the devs haven't fixed it. For now you can use the subquery I wrote to actually graph the metrics but the growth of the number of time series is enough to make me drop the metric entirely until a fix is made. We may try to deploy a copy of the code that removes those labels.

#3670

#3671