Telemetry
Hardware telemetry - what was happening on the GPU server while a job ran.
Accessing
Telemetry is accessible from the job detail page under Dashboard -> Monitoring (/dashboard/monitoring). The left panel is a job selector; the main panel renders charts for the selected job.
What's captured
| Metric | Source |
|---|---|
| GPU power (W) | NVIDIA NVML / ROCm SMI / TPU runtime |
| GPU utilization (%) | Same |
| GPU memory (%) | Same |
| CPU utilization (%) | OS-level |
| System RAM (%) | OS-level |
| Per-GPU breakdown | When multiple GPUs per server |
TPU jobs use a parallel TPU-specific telemetry path; surfaced via v_tpu_telemetry_job_summary. The agent collects these metrics from LibTPU, Google's TPU runtime library, which exposes hardware counters for per-chip duty cycle, tensor core utilization, and HBM capacity directly through its metrics endpoint. LibTPU provides these readings at sub-second granularity, allowing the agent to capture the same telemetry fidelity as NVML on GPU workloads. For the full metric list, units, and how each is collected, see System Metrics in the methodology.
What to look for
- GPU power flat at TDP: fully utilizing the hardware. Good for benchmark runs.
- GPU power oscillating: likely batching dynamics; usually fine.
- GPU utilization
< 50%at high concurrency: workload is bottlenecked elsewhere (CPU preprocessing, network, memory bandwidth). - High CPU during VLM jobs: image preprocessing path. Try a higher-core-count server.
Telemetry summary card
For a quick at-a-glance, the right side of the panel summarizes mean / peak / min for each metric over the job's duration. Useful when you want the headline number without scrubbing a chart.