Code

Telemetry implementation#2359

Code

Closed

djsaunde wants to merge 22 commits intomainaxolotl-ai-cloud/axolotl:mainfrom telemetryaxolotl-ai-cloud/axolotl:telemetryCopy head branch name to clipboard

+2,399-41

Commits

Commits on Feb 26, 2025

• initial telemetry manager impl

  djsaunde

  committed
  5220e8c

  Copy full SHA for 5220e8c

• adding todo

  djsaunde

  committed
  76d951a

  Copy full SHA for 76d951a

• updates

  djsaunde

  committed
  07ffd47

  Copy full SHA for 07ffd47

• updates

  djsaunde

  committed
  c506103

  Copy full SHA for c506103

• progress on telemetry: config load, process, model load, train start / end, error tracking

  djsaunde

  committed
  3076b8d

  Copy full SHA for 3076b8d

• update error file path sanitization function; adding more error tracking

  djsaunde

  committed
  ec36839

  Copy full SHA for ec36839

• updated sanitization logic, tests

  djsaunde

  committed
  7927abf

  Copy full SHA for 7927abf

• adding runtime metrics (cpu + gpu memory, steps/s, etc.)

  djsaunde

  committed
  3760175

  Copy full SHA for 3760175

• tests for runtime metrics telemetry and assoc. callback

  djsaunde

  committed
  fd5d5ae

  Copy full SHA for fd5d5ae

• small update / fix

  djsaunde

  committed
  e373a6b

  Copy full SHA for e373a6b

• simplifying path redaction

  djsaunde

  committed
  b23187d

  Copy full SHA for b23187d

• sleep on all ranks in distributed setting

  djsaunde

  committed
  675b65d

  Copy full SHA for 675b65d

• adding back in base_model redaction w/ whitelist

  djsaunde

  committed
  d3d63c1

  Copy full SHA for d3d63c1

• fix

  djsaunde

  committed
  c9af72c

  Copy full SHA for c9af72c

• doc update

  djsaunde

  committed
  49ac79e

  Copy full SHA for 49ac79e

• improved redaction, send system info during model config load telemetry, etc.

  djsaunde

  committed
  5afad67

  Copy full SHA for 5afad67

• adding runtime metrics / system info additional accelerator support, etc.

  djsaunde

  committed
  aea0e76

  Copy full SHA for aea0e76

• adding runtime metrics / system info additional accelerator support, etc.

  djsaunde

  committed
  61266ab

  Copy full SHA for 61266ab

• remove duplicate info

  djsaunde

  committed
  09611ce

  Copy full SHA for 09611ce

• fixes

  djsaunde

  committed
  cae8c76

  Copy full SHA for cae8c76

• fix issue with tests in ci

  djsaunde

  committed
  07bb418

  Copy full SHA for 07bb418

• distributed fix

  djsaunde

  committed
  f144319

  Copy full SHA for f144319