Skip to main content

Telemetry

Hardware telemetry - what was happening on the GPU server while a job ran.

Accessing​

Telemetry is accessible from the job detail page under Dashboard -> Monitoring (/dashboard/monitoring). The left panel is a job selector; the main panel renders charts for the selected job.

What's captured​

MetricSource
GPU power (W)NVIDIA NVML / ROCm SMI / TPU runtime
GPU utilization (%)Same
GPU memory (%)Same
CPU utilization (%)OS-level
System RAM (%)OS-level
Per-GPU breakdownWhen multiple GPUs per server

TPU jobs use a parallel TPU-specific telemetry path; surfaced via v_tpu_telemetry_job_summary. The agent collects these metrics from LibTPU, Google's TPU runtime library, which exposes hardware counters for per-chip duty cycle, tensor core utilization, and HBM capacity directly through its metrics endpoint. LibTPU provides these readings at sub-second granularity, allowing the agent to capture the same telemetry fidelity as NVML on GPU workloads. For the full metric list, units, and how each is collected, see System Metrics in the methodology.

What to look for​

  • GPU power flat at TDP: fully utilizing the hardware. Good for benchmark runs.
  • GPU power oscillating: likely batching dynamics; usually fine.
  • GPU utilization < 50% at high concurrency: workload is bottlenecked elsewhere (CPU preprocessing, network, memory bandwidth).
  • High CPU during VLM jobs: image preprocessing path. Try a higher-core-count server.

Telemetry summary card​

For a quick at-a-glance, the right side of the panel summarizes mean / peak / min for each metric over the job's duration. Useful when you want the headline number without scrubbing a chart.