[Feature]: rocm-smi execution in the loop and polling the GPU every few seconds (parameter -l)

[vedranmiletic]

Suggestion Description

nvidia-smi supports-l <seconds> and -lms <milliseconds> parameters, which keep the tool running and poll the GPU for new information every <seconds> or <milliseconds> specified. Similar behavior can be achieved with watch -n <seconds> nvidia-smi, but -l <seconds> has less overhead on the measured system.

It would be nice to support this feature in rocm-smi as well.

Operating System

RHEL

GPU

MI300A

ROCm Component

rocm-smi

[dmitrii-galantsev]

We have this in amdsmi:
amd-smi monitor -w 1 ->

 TIMESTAMP  GPU  POWER  GPU_TEMP  MEM_TEMP  GFX_UTIL  GFX_CLOCK  MEM_UTIL  MEM_CLOCK  ENC_UTIL  DEC_UTIL    VCLOCK    DCLOCK  SINGLE_ECC  DOUBLE_ECC  PCIE_REPLAY  VRAM_USED  VRAM_TOTAL     PCIE_BW  PVIOL  TVIOL  TVIOL_ACTIVE  PHOT_TVIOL  VR_TVIOL  HBM_TVIOL  GFX_CLKVIOL
1740686195    0    8 W     28 °C     32 °C       0 %      0 MHz       0 %     96 MHz       N/A     0.0 %     0 MHz     0 MHz           0           0            0      16 MB    30679 MB    N/A Mb/s    N/A    N/A           N/A         N/A       N/A        N/A          N/A
1740686195    1    8 W     29 °C     30 °C       0 %      0 MHz       0 %     96 MHz       N/A     0.0 %     0 MHz     0 MHz           0           0            0      16 MB    30679 MB    N/A Mb/s    N/A    N/A           N/A         N/A       N/A        N/A          N/A


 TIMESTAMP  GPU  POWER  GPU_TEMP  MEM_TEMP  GFX_UTIL  GFX_CLOCK  MEM_UTIL  MEM_CLOCK  ENC_UTIL  DEC_UTIL    VCLOCK    DCLOCK  SINGLE_ECC  DOUBLE_ECC  PCIE_REPLAY  VRAM_USED  VRAM_TOTAL     PCIE_BW  PVIOL  TVIOL  TVIOL_ACTIVE  PHOT_TVIOL  VR_TVIOL  HBM_TVIOL  GFX_CLKVIOL
1740686196    0    8 W     28 °C     32 °C       0 %      0 MHz       0 %     96 MHz       N/A     0.0 %     0 MHz     0 MHz           0           0            0      16 MB    30679 MB    N/A Mb/s    N/A    N/A           N/A         N/A       N/A        N/A          N/A
1740686196    1    8 W     29 °C     30 °C       0 %      0 MHz       0 %     96 MHz       N/A     0.0 %     0 MHz     0 MHz           0           0            0      16 MB    30679 MB    N/A Mb/s    N/A    N/A           N/A         N/A       N/A        N/A          N/A