dotnet monitor in kubernetes cluster when 2 or more replicas

[Iceforestik]

I want to monitor several pods of the same application in kubernetes deployment. It turns out to monitor one pod correctly, and if there are many of them, then how to monitor correctly

single replicas deployment config

apiVersion: apps/v1
kind: Deployment 
namespace: test       
spec:
  progressDeadlineSeconds: 600
  replicas: 1 
  template:  
    spec:
      volumes:
      - name: diagnostics
        emptyDir: {}
      containers:
      - env:
        - name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
          value: Listen
        - name: DOTNETMONITOR_DiagnosticPort__EndpointName
          value: /diag/port.sock
        - name: DOTNETMONITOR_Storage__DumpTempFolder
          value: /diag/dumps
        - name: DOTNETMONITOR_Urls
          value: http://localhost:52323           
        image: manager:master
        imagePullPolicy: Always
        name: chart
        ports:
        - containerPort: 10800
          protocol: TCP
        securityContext: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
          - mountPath: /tmp
            name: diagnostics
      - name: sidecar
        image: dotnet/nightly/monitor
        ports:
        - containerPort: 52325
        args: ["collect","--urls", "http://*:52323", "--metricUrls", "http://*:52325", "--no-auth"]
        volumeMounts:
          - name: diagnostics
            mountPath: /tmp
        resources:
          limits:
            cpu: "1"
            memory: 500Mi
          requests:
            cpu: 200m
            memory: 500Mi

[jander-msft]

To monitor multiple pods, you can still use dotnet-monitor as a sidecar in each pod. The downside is that you now have one dotnet-monitor for each pod and must know how to connect to each one (e.g. which dotnet-monitor ports correspond to which pod).

Another way to do this is to create a DaemonSet that contains the dotnet-monitor container and use hostPath volumes to share the diagnostic communication between the target containers and the dotnet-monitor container. This would allow you to use one dotnet-monitor per node that can monitor containers across different pods. One current drawback to this approach is that the /metrics route will only serve up metrics for a single process, so once you introduce multiple processes that may match the default process specification, then the route will not report anything. There is an outstanding request to support this ability.