1. NVIDIA GPU Fan - Pwr:Usage - ERR!

root@yongqiang:~# nvidia-smi Mon Nov 25 17:38:06 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.43 Driver Version: 418.43 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce RTX 208... On | 00000000:1B:00.0 Off | N/A | |ERR! 39C P0 ERR! / 250W | 0MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce RTX 208... On | 00000000:1C:00.0 Off | N/A | | 22% 31C P8 24W / 250W | 34MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 GeForce RTX 208... On | 00000000:1D:00.0 Off | N/A | | 22% 29C P8 3W / 250W | 0MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 GeForce RTX 208... On | 00000000:1E:00.0 Off | N/A | | 22% 29C P8 11W / 250W | 2279MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 4 GeForce RTX 208... On | 00000000:89:00.0 Off | N/A | | 22% 29C P8 21W / 250W | 2387MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 5 GeForce RTX 208... On | 00000000:8A:00.0 Off | N/A | | 22% 28C P8 21W / 250W | 537MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 6 GeForce RTX 208... On | 00000000:8B:00.0 Off | N/A | | 22% 25C P8 1W / 250W | 209MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 7 GeForce RTX 208... On | 00000000:8C:00.0 Off | N/A | | 22% 24C P8 18W / 250W | 10MiB / 10989MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 1 55182 C detection_worker_gpu1 12MiB | | 1 55203 C detection_worker_gpu1 12MiB | | 3 54733 C detection_worker_gpu3 1136MiB| | 3 54752 C detection_worker_gpu3 1132MiB | | 4 54823 C detection_worker_gpu4 1188MiB | | 4 54842 C detection_worker_gpu4 1188MiB | | 5 55057 C detection_worker_gpu5 514MiB | | 5 55282 C detection_worker_gpu5 12MiB | | 6 55082 C detection_worker_gpu6 199MiB | +-----------------------------------------------------------------------------+ root@yongqiang:~#

2. NVIDIA GPU 硬件故障

GPU 0 风扇转速 (Fan) 和功耗 (Pwr:Usage) 均显示为ERR!。

2.1. 检查物理供电线缆

关机并断开电源。打开机箱，拔掉 GPU 0 上的 PCIe 供电线。检查接口是否有烧焦、熔化或变色迹象。重新用力插紧供电线，确保卡扣锁死。

2.2. 重新插拔显卡

PCIe 插槽接触不良会导致数据传输错误。将 GPU 0 从主板上拆下。清理 PCIe 金手指和主板插槽的灰尘，重新紧固地插回插槽。

2.3. 重启服务器

重启服务器，直接运行sudo reboot。

2.4. 调换插槽与线缆 (交叉测试)

• 将 GPU 0 和 GPU 1 的供电线互换
• 将 GPU 0 和 GPU 1 的 PCIe 插槽互换

如果错误跟随 GPU 0 移动，说明是显卡硬件损坏。如果错误留在原插槽或原线缆，说明是主板插槽或电源线材故障。

请联系供货方技术支持。
对于 AMAX 供货方来说是已知问题，换卡是目前已知的方案。

References

[1] Yongqiang Cheng (程永强), https://yongqiang.blog.csdn.net/
[2] NVIDIA GPU Fan - Pwr:Usage - ERR!, https://yongqiang.blog.csdn.net/article/details/103244347